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ABSTRACT 

This study sought to evaluate the goals and functions 
of practical tests for undergraduate students in the natural sciences 
in university distance education (DUE) programs in the Netherlands, 
and to compare the findings with results from similar research 
conducted in regard to science practicals at the Open University of 
the Netherlands (OuN) and traditional university-level education 
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specific end-terms for undergraduate practicals in the natural 
sciences. The study found that most faculty at DUE institutions 
approached practicals in similar ways, and that the ability to solve 
problems, interpret experimental data, and use knowledge and skills 
in unfamiliar situations were considered the most important general 
objectives for students at DUE institutions. It also found that 
although the differences between OuN and DUE rankings were larger 
than the difference between DUE and UE rankings, the rankings by 
faculty at each of the three institutions were statistically unique. 
Sixteen appendixes provide data on the ratings of general objectives, 
specific objectives, and end-terms by DUE, UE, and OuN faculty. 
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Summary 

The research described in this report is part of a research project on the perceived importance of 
learning objectives and end-terms for practicals in undergraduate higher science education. This 
report focuses on higher distance education. The results for institutes for university distance 
education (DUE) are reported and compared with the results obtained in earlier studies at the 
Open university of the Netherlands (OuN) and the more traditional (face-to-face) universities in 
the Netherlands (UE). 

A focal point of this study is to determine whether, and to what extent, the perceived importance 
of the objectives and end-terms depends on the type of institute (traditional face-to-face education 
or open distance education) and/or the type of education offered (monodisciplinary versus 
inter/multidisciplinary). 

An inventory of learning objectives and end-terms for undergraduate practicals in the natural 
sciences, as contained in the literature, was made at an earlier stage of this project The resulting 
list, consisting of a small number of general learning objectives and end-terms (and a large 
number of specifications thereof), is the starting point of the instruments used in this and 
previous research. 

The respondents in this study are members of the faculties and/or departments of distance 
universities offering education in the natural sciences throughout the world. 

The major conclusions from the research are: 

- The different institutes making up the sample population for DUE approach practicals in the 
Natural Sciences in a similar way. This can be seen on the coefficient of concordance (Kendall's 
W). The degree of concordance across the sample population is large. As a matter of fact it's 
level of significance is about the same as that of the OuN. This is rather remarkable 
considering the differences between the individual respondents with respect to culture, 
language, educational background, etcetera. The only things which they all appear to have in 
common is that they all are institutes for distance education and they all approach the Natural 
Sciences in a monodisciplinary way. 

- The ability to solve problems, interpret experimental data and use knowledge and skills in 
unfamiliar situations are far and away the most important general objectives for the students 
attending these institutions to achieve. The importance of the first two is also reflected in the 
ratings of their component specific objectives. The operationalisations of the third general 
objective are, on the other hand, ranked very low. 

- The ability to describe an experiment is far and away the least important general objective for 
the students attending these institutions to achieve. When the component specific objectives 
subsumed under the general objectives are used to calculate the relative importance of the 
general objectives, then the ability to describe an experiment jumps from last to second place. 

- The specific objectives which DUE rates as indispensable are primarily elementary laboratory 
skills relating to doing practical work. Those objectives which were rated as neutral are, on the 
other hand, primarily higher academic skills relating to reasoning and problem solving. 
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The Open university of the Netherlands and other institutions for higher distance education 

This report is part of a research project into the evaluation of learning objectives and end-terms 
for undergraduate science practicals in institutes of higher education. Two reports 'Learning 
objectives for practicals at the Open university of the Netherlands' and 'Learning objectives for 
practicals in institutes for higher education in the Netherlands' have preceded the present study. The 
former is a study of the value placed upon objectives and end-terms for practicals at the Open 
university of the Netherlands (OuN). The latter concerns the evaluation of (the same) objectives , 
and end-terms by more 'traditional* institutes of higher (face-to-face) education in the 
Netherlands, both university education (UE) and higher vocational education (HVE). 
In the present report we discuss the evaluation of these same learning objectives and end-terms by 
a number of institutes for university distance education (DUE) which have programmes in the 
Natural Sciences and compare these results with those found at the OuN and in UE. 

Traditional (face-to-face) universities differ so much from distance universities (DU's) that it is 
quite conceivable that a number of objectives and end-terms for practicals to which they strive 
differ from each other. This assumption is not so much based on the fact that DU*s such as the 
OuN have limited possibilities for organizing practicals, but rar>er on the special nature of DLPs, 
and specifically their underlying philosophies (distance education instead of face-to-face 
education). Apart from this difference, the OuN also differs from most other universities (both 
distance and face-to-face) in that it offers a different type of science education projrjramme (an 
inter- or multidisciplinary programme as opposed to a monodisdplinary one). These differences 
have led to a reconsideration of learning objectives and end-terms for science education in general 
md science practicals in particular within the OuN. 

With regard to the nature of the institutions, it can be said that the OuN as well as the distance 
universities in other countries distinguish themselves from other types of institutes in that they 
put special emphasis on a number of freedoms which they offer to students. They all allow their 
students to study where they wish, while some (Le. the OuN, the Center for Distance Education at 
the Empire State College) also allow their students to study when they want and to choose their 
own pace of study. As a consequence, the number of times that students should be required to 
come to a specific place at a specific time in a specific phase of their study, e.g. for practicals, 
need be kept to a minimum. This limitation should necessitate these institutions to choose those 
learning objectives and end-terms that are really necessary for their programmes from all the 
possible learning objectives for a specific domain of study. This, in turn, implies that these 
institutions should reconsider the role and function of practicals in the undergraduate part of 
their educational programmes and give priority to a certain type of learning objectives (e.g. 
academic or cognitive skills) at the expense of other learning objectives (e.g. motor skills). 

With regard to the type of educational programme offered, it can be said that the OuN (as well as 
some fairly new disciplines within UE) offers multi- or interdisciplinary programmes. A typical 
example of such an educational programme is the Environmental Science programme at the OuN, 
which seeks an integration of the natural and the social sciences. All these programmes aim at an 
integration or synthesis of disciplines by putting special emphasis on evaluative or synthetic 
behaviour and by focusing on the syntactic structure of the various disciplines. In 
monodisciplinaiy programmes, on the other hand, the emphasis will be on analytic behaviour and 
on the substantive structure of a specific discipline. 
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Together these differences yieJd a 2 x 2 matrix, of which the cells can be compared with each 
other: 



monodisciplinary 

inter-/multi- 
disciplinary 



distance 
education 



face-to-face 

university 

education 



most open 
distance universities 


tnostUE 
science faculties 


OuN 


new UE 
science faculties 



The present report is the third and last in a series of reports concerning the empirical part of the 
research project 'Practicals in Higher Science Education 9 . The first report (Meester, Kirschner, 
Middelbeek & Hermans, 1989; Kirschner, Meester & Middelbeek, 1990) discusses the results of a 
research project carried out within the staff of the product group Natural Sciences at the OuN on 
the perceived importance of learning objectives and end-terms for practical at the OuN. 
The second report (Meester, Kirschner, Middelbeek & Hermans, 1990) discusses the results of 
research into the appreciation of learning objectives for practicals in other institutions of higher 
education (both university and polytechnic or vocational colleges) in the Netherlands apart, as 
well as in comparison with the OuN. Paragraph 2.1 gives a short synopsis of the results of this 
research. 

This final report discusses the results of research into learning objectives and end-terms for 
practicals in institutes of higher distance education abroad in comparison with the OuN and 
traditional (Dutch) universities. 



2 The research 

This research is aimed at the evaluation of general and specific learning objectives and end-terms 
for undergraduate science practicals in different institutes of university level distance education. In 
the analysis, the following stages can be distinguished: 

- arrangement in order of importance of general learning objectives for practicals in DUE; 

- evaluation of specific learning objectives for practicals by the 
respondents (and their relationship to the six general learning objectives); 

- evaluation of end-terms for practicals by the respondents; 

- comparison of the results of DUE with those of UE; 

- comparison of the results of the OuN with those of DUE and UE; 

- explicit formulation of the similarities and differences between the three types of education on 
the basis of their views on undergraduate practicals. 

Before we start to describe the actual research, we will define a number of concepts used 
throughout this report. 

The concept of practical is used to indicate a didactic method for learning and practising all the 
activities involved in the carrying out of one's profession. Within the Natural Sciences at the 
university level, this professional practise entails experimental work, beginning with the 
conception of a problem or the observation of a phenomenon to the communication of finding* 
in the form of a report or a presentation. Hodson (1988) presents this relationship as follows: 
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experimental work is a subset of laboratory work. He refers to this as laboratory bench work. 
Laboratory work is a subset of practicals, which in turn is a subset of the didactics of science 
education. Figure 1 gives a schematic representation of this interrelationship. 




Figure 1 

Interrelationship between experiments, laboratory work and practicals (Hodson, 1988) 

The undergraduate part of an educational programme is the part of the programme that consists 
of short, well-defined courses that are obligatory for all students (of a certain discipline). In 
traditional universities in the Netherlands, the biology, chemistry and physics programmes 
comprise approximately 800-1000 hours of undergraduate practicals. Besides this undergraduate 
part of the programme, approximately 1200 hours are reserved for work experience or research 
training. 

The general learning objectives have been formulated on the basis of the expected learning results; 
they have been fitted with a list of achievements that specify the eventual behaviour of the 
students (the specific learning objectives). 

The difference between objectives and end-terms for practicals can be described as follows: for the 
attainment of the general and the specific objectives, the execution of practicals is a purpose in 
itself whereas for the achievement of the end-terms, the execution of practicals (besides many 
other didactic methods) is a means to an end. 



Synopsis of the results of the research within the OuN and other institutions for higher education in 
the Netherlands 

In this section, we discuss the most poignant results of the previous two studies. For more 
complete information, the reader is referred to these two reports as well as appendices 1 through 
4 in which the scores for the specific objectives and end-terms for the OuN and UE are given. 

General learning objectives 

The members of the product group Natural Sciences of the OuN and science faculty members 
from Dutch universities and institutes of higher vocational education were asked to evaluate 
(using a paired comparison evaluation instrument) eight general learning objectives (Kirschner 
and Meester, 1988). The general objectives were formulated on the basis of a clustering of the 
specific objectives that were found in the literature and reflect the course of an ideal experiment. 
In the course of the research, as a result of further analysis and reclassification of the specific 
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objectives, two general objectives were excluded so that six were left 1 The results of these 
evaluations are listed in table 1. 



Table 1 

Ranking of importance of general learning objectives 
general learning objectives? 



to solve problems (A) 

to use knowledge and skills in unfamiliar situations (B) 
to design (simple) experiments to test hypotheses (C) 
to use lab skills in performing (simple) experiments (D) 
to interpret experimental data (E) 
to clearly describe the experiment (F) 

a The letters after the objectives correspond to the letters in appendix 8. 
b 1 m most important; 6 * least important 
c p < 0.001 



rant* 


ave 


rank 


ave 


OuN-UE 


OuN 


score 


UE 


score 




4 


2.83 


4 


255 


028 


1 


3.58 


2 


2.73 


0.85 


3 


3.08 


1 


3.09 


-0.01 


6 


0.50 


5 


2.05 


-1.55 c 


2 


3.42 


3 


2.68 


0.74 


5 


1.42 


6 


1.77 


-035 



The difference between the most and least highly valued objective at the OuN is 3.08. A 'paired u 
test* has demonstrated that a distinction can be made between the four most highly valued general 
objectives for the OuN (B 9 E, C, and A) which do not differ from each other statistically (f=0.68; 
Z^O.12) and the two least valued general objectives (F and D) (f=2.2; p<0.005). 
The order of priority with regard to the general objectives as expressed by the respondents from 
UE is, on the other hand, quite subtle. The difference here between the most and least important 
objective is only 132. Two important general objectives *to design experiments to test hypotheses' 
(C) and *to interpret experimental data* (E) score significantly higher (/=3.23; /><0.05; /=234; 
/><0.05 respectively) than the least important general objective *to clearly describe an experiment' 
(F). The most important objective (C) also scores significantly higher (/*238; p<0.05) than the 
objective 'to use laboratory skills'^). Otherwise there are no significant differences. 
The last column of the table gives the differences between the average scores of the OuN and UE. 
A positive difference means that the objective is considered more important at the OuN than in 
UE. A negative difference means that the opposite is the case. 

These results tend to the conclusion that, with regard to the learning objectives for undergraduate 
practical, the OuN places much more emphasis on learning to use knowledge in unfamiliar 
situations (B) and learning to interpret data (£) than UE. This is probably the result of the highly 
specific character of the OuN which focuses on and strives to provide 'a type of education which 
enables its graduates to learn to think in a problem oriented manner (End-terms, 1986). On the 
other hand, LIE places a great deal more emphasis on learning to use laboratory skills in 
performing experiments (D) than the OuN. In other words, what the OuN graduate misses in 
laboratory skills is/must be more than compensated by training in problem oriented and problem 
solving thinking. None of the other differences was significant 

Specific learning objectives 

We used a Likert-scale inventory to assess the perceived importance of 64 specific learning 
objectives. The frequency distribution of the specific objectives in order of decreasing importance 
can be found in table 2. 



1 The motivation for this decision can be found in paragraph S.l 
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Table 2 



Frequency distribution and percentages of 64 specific learning objectives arranged in order of 
decreasing importance 



type of 
institute 



2 



3 



4 



5 



UE 
OuN 



23(36%) 
11 (17%) 



33 (52%) 8 (13%) 
37 (58%) 10 (16%) 



6(9%) 



a 1 = indispcnsaMc (1.00 * x < 1.80); 2 * Important (1.80 * x < 2.60); 3 - neutral (2.60 * x < 3.40); 
4 * not really necessary (3.40 * x < 4.20); 5 » wperfluow (4.20 * x * 5.00) 

A striking aspect of this table is the large number of objectives classified as worthwiiile 
(indispensable or important) by both respondent groups, especially by the group of UE 
respondents (88% of all the specific learning objectives), whereas the categories not really 
necessary and superfluous are empty for UK This is not really surprising, given the fact that the 
list of specific objectives is based on literature on objectives for practicals in traditional 
undergraduate science education (Kirschner and Meester, 1988). The OuN, on the other hand, 
assesses nearly 10% of the objectives as being not really necessary and is less positive about twice 
the number of objectives as UE (16 vs. 8 objectives). This is also not surprising when one 
considers the programmatic and philosophical differences between the OuN and UE Five of these 
objectives are common to both response groups. Nine of the remaining eleven objectives deal 
with using laboratory skills in performing experiments (general objective D). This supports the 
differences noted in the previous section. 

This quantitative analysis was enriched with the following qualitative analysis. The OuN varies 
greatly from UE with regard to 21 specific objectives. These objectives provide specific 
information about how the OuN differs from UE. These may be objectives which the OuN 
considers much more important than UE (AZ "as 1) or objectives which OuN values as much less 
important than UE (AZ s -1). It does not necessarily mean that UE does not value these 
objectives as important, but merely that the OuN values them as either much more important or 
much less important. A typical example of this is the objective 'to confirm facts, principles and 
theory from lecture/books* (see table 3). 

The learning objectives of the list with negative AZ values all, except for one, relate to practical 
manual skills during the actual realization of an experiment All of them, except for 'processing of 
experimental data* are specifications of general objective D 'to use laboratory skills in performing 
(simple) experiments*. UE values these practical laboratory skills as indispensable objectives, but 
for an OuN graduate in science they are not really necessary (Meester et al., 1989). 
Of the 11 objectives with positive iz-scores, seven are specifications of general objective^ 'to 
solve problems*. 
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Table 3 



Specific learning objectives with a large difference between the OuN and UE 



specific learning objective 



AZ(OuN-UE) 



general 
obj. 



to incorporate unexpected experimental results in a 
new model 

to solve problems in a multi-solution situation 

to derive and evaluate relationships 

to solve difficult scientific problems 

to confirm facts, principles and theory from lecture/books 

to derive testable hypotheses from theories 

to recognize and define scientific problems 

to be flexible in modifying experiments 

to discuss results with other scientists 

to react adequately to unforeseen results 

to decompose large problems into smaller problems 



2.14 
157 
1.46 
137 
135 
134 
1.19 
1.12 
1.09 
1.09 
1.04 



E 
A 
A 
A 
A 
A 
B 
A 
F 
B 
A 



to use practical (as opposed to theoretical) lab skills 

to calibrate instruments 

to set up lab equipment quickly and correctly 

to collect experimental data 

to carry out accurate measurements 

to handle modern equipment 

to put basic laboratory techniques to use 

to conduct experiments safely 

to manipulate apparati 

to process experimental data 



-1.86 
-1.74 
-1.73 
-1.72 
-1.70 
.1.66 
.1.47 
.1.47 
-1.46 
-1.02 



D 
D 
D 
D 
D 
D 
D 
D 
D 
E 



AZ * +\ means this learning objective is considered much more important by the OuN than by UE 
AZ * -1 means this learning objective is considered much less important by the OuN than by UE 



End-terms 

The article by Kirschner & Meester (1988) mentions the following two general end-terms: 

I to obtain good (scientific) attitudes 

II to understand the scientific method 

Each of these end-terms corresponds to a large number of specifications. Nearly all of the 
specifications of the end-terms were rated as important to very important by the product group 
Natural Sciences of the OuN. This is not surprising considering the fact that these are end-terms 
for an undergraduate study in the natural sciences as a whole, for which practicals are but a 
means to that end. 

When applying the criterion that a \AZ\ * 1 indicates an essential difference between the 
institutes to be compared, the OuN differs considerably from UE with regard to six end-terms. 
Three of the end-terms were rated higher by the OuN than UE, namely: to be self-confident and 
independent, to build a framework for facts, principles and theory from lectures/books, and to 
survey literature relevant to a problem. Three other end-terms were rated more important by UE 
than by the OuN, namely: to do experiments, to work in research an* development labs, and to 
use the laboratory as an instrument of discovery. These differences confirm the picture that UE is 
more concerned with mastering the actual techniques of experimenting while the OuN is more 
interested in the theoretical side of practicals. 

One striking aspect is the difference with respect to the end-term 'to be self-confident and 
independent 9 which the OuN rates much more highly than UE. UE students, by virtue of their 
admission to the university are already fairly self-confident and independent OuN students must 
accrue these qualitk" through their studies. The staff there is also well aware of this and as such 
places more value on uch end-terms. 
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Finally, the three end-terms rated more highly by UE relate to the using and carrying out 
experiments and research. The OuN has to a large extent abandoned this type of activity when 
programming its Natural Science distance education. The end-term to build a framework for 
facts, principles and theory from lectures/books* indicates that within the OuN the emphasis is on 
integration of (theoretical) knowledge from various sources. 

In conclusion, the results of the first two studies showed that: 

. There is a definite difference between the OuN and UE. This difference is noteworthy, but 
explainable. Although the OuN and UE are both academic (as opposed to vocational) 
institutions, this similarity is not as strong as the differences between them. UE is 
monodisciplinary, the OuN is not UE trains its students to do research; the OuN trains its 
students to apply the research of others. 

- UE values the 'use of laboratory skills in performing (simple) experiments' (D) more highly 
than the OuN does. This was to be expected. 

- The identity of the type of institute clearly comes to the fore. UE places little emphasis on 
practical skills (although much more than the OuN), but emphasizes the formulation of 
hypotheses or models and the design of experiments to test hypotheses (academic skills). The 
OuN attaches great importance to the ability to define and solve problems, to derive 
hypotheses from theories and to adapt models to new experimental results, and puts much less 
emphasis on laboratory skills (even less than UE). 



3 Respondents 

For this part of the research, 36 faculties, sectors or departments of distance universities offering 
education in (a subject area of) the Natural Sciences were written to with the request to let two 
instruments (see paragraph 4) be filled in by the person or persons responsible for practical 
within the institute. 

The institutes concerned were solicited according to the following two-step procedure: 
. the boards of governors/directors of the various faculties, sectors, departments etc were written 
to with the request to pass the instruments sent to them on to either a member of the educational 
committee or the coordinator of practicals of the group; 

- in addition, the boards of governors/directors were asked to make the names and addresses of 
the respondents known to us by means of a reply card and a stamped self-addressed envelope, to 
allow us to contact him/her personally, if necessary. 

Of the 36 institutes that were approached, three sent the instruments back blank for reasons of 
having no science program or teaching science at a lower level. 

Of the remaining 33 institutes 18 respondents representing 11 different universities returned the 
inventories which leaves us with a response of 33%. Of these 18 respondents, three are left out of 
consideration in this study because they didn't meet our condition of being concerned with 
practicals in science education at an institute of higher distance education. The list of universities 
participating in this study can be found in appendix 5. 

The remaining 15 respondents (10 institutes) represent the science disciplines chemistry (5), 
biology (4), physics (3), geology and earth sciences (2). Most of the respondents have a long 
record of service in higher distance education: ten of them have been working at these institutes 
for more than 10 years. 

We decided to take as a starting point in this report the number of respondents instead of 
responding universities because of the fact that the participants belonging to the same institute 
appeared to respond for different disciplines within these institutes. 
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4 Research method 

4.1 Instruments 

Two instruments (inventories) were developed to determine the perceived importance of the 
learning objectives and end-terms. The first instrument consists of a paired comparison of eight 
general learning objectives (see appendix 6). These eight general objectives, derived from the 
article by Kirschner & Meester (1988), are based on the successive steps that a scientist' ideally 
takes when carrying out an experiment 
These are: 

- to formulate hypotheses 

- to solve problems 

- to use knowledge and skills in unfamiliar situations 

- to design (simple) experiments to test hypotheses 

- to use laboratory skills in performing (simple) experiments 

- to interpret experimental data 

- to clearly describe the experiment 

- to remember the central idea of an experiment over a significantly long period of time. 

In the instrument, each of the eight objectives is paired with each of the other seven, yielding a 
total of 28 pairs of objectives. Each of these pairs is presented on a separate page of a 
questionnaire book with a request to the respondents to indicate which of the two objectives, they 
consider most important 

An example of a page: 

'After completing their study in the natural sciences, students should be able to: 

0 use knowledge and skills in unfamiliar situations 
0 interpret experimental data' 

The pairs were arranged in such a way that each objective appeared an equal number of times as 
the first and as the second objective. The objectives were distributed as equally as possible over 
the inventory, although it turned out to be impossible to prevent the same objective from 
appearing in two successive pairs. The respondents were requested to start by going through the 
inventory once before filling it in and, once started, not to leaf forward or backward so that they 
would not be influenced by previously made choices. 

We deliberately opted for a comparison in pairs instead of introducing ranking of priority 1 - 8, as 
has been done in several other studies (Kerr, 1963; Woolnough, 1967; Gould, 1978; Boud, 1980; 
Beatty, 1982). In our view, it is very difficult to rank the eight learning objectives as such, because 
the differences in importance are usually fairly small. A paired comparison forces the respondent 
to weigh the objectives in each pair against each other, and to choose between them. 

The second instrument comprises, in random order, the specific learning objectives and end-terms 
taken from the same article by Kirschner & Meester. The respondents were asked to indicate the 
importance of each objective on a five-point Likert scale (see appendix 7). 
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The five scale units range from indispensable to superfluous and have been given the following 
meaning: 

indispensable: This learning objective is essential and must be included in the program; 

much emphasis should be placed on this objective 
important: This objective should be included but not necessarily emphasised 

neutral: I don't have an opinion as to this objective; by a vote on such an 

objective I would abstain 
not really necessary: This objective is of minimal importance; if there is a lack of time or 

opportunity then this objective need not be included 
superfluous: This objective should not be included in the curriculum; no time need be 

reserved for this objective 

The respondents were again asked to start by reading through all the items once before filling 
them in and, once started, not to leaf forward or backward or to change answers already given. 

An undergraduate study in the natural sciences should prepare students 
to: 



handle modern equipment 



indispensable 


important 


neutral 


not really necessary 


superfluous 


- understand the scope and limiting conditions of the experimental 


techniques used 








indispensable 


important 


neutral 


not really necessary 


superfluous 


- analyse experimental data in order to draw conclusions from them 


indispensable 


important 


neutral 


not really necessary 


superfluous 



Initially, the 97 specific objectives and end-terms of the second instrument were formulated in the 
same way as in the article of Kirschner & Meester (in random order). On closer consideration - 
after a draft version of this instrument had been presented to a number of scientists, teaching 
methodologists and educational experts - several items were found to be vague or open to more 
than one interpretation. After reformulation, the list of specific learning objectives and end-terms 
was submitted to a number of educational experts and natural scientists for evaluation. As a result 
of this, five items were excluded, six items were split up into two items, two items were split up 
into three items and many items were reformulated in order to render them as unequivocal as 
possible. Some items also changed places: a few specific objectives became end-terms and vice 
versa. The final list consists of 102 items, among which 64 specific learning objectives and 38 
specific end-tenn*. This list was sent to the institutes, as described in paragraph 3. 
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4.2 Validity and reliability 

A test or scale is valid (or invalid) for the scientific or practical purpose of its users and not valid 
(or invalid) in a vacuum. The inventories developed here are neither meant for prediction nor for 
the testing of hypothesized relations or theoretical constructs. Criterion-related validity and 
construct validity respectively are therefore not of consequence here. What is of consequence is the 
representativeness of sampling adequacy of the content of a measuring instrument Content validity 
is guided by the question: "Is the substance or content of this measure representative of the 
content or the universe of content of the property being measured 91 (Kerlinger, 1973). Seeing as 
how the inventories are based upon, to our knowledge, the most complete set of objectives 
collected to date (Kirschner & Meester, 1988), the answer to this question is an unequivocal yes. 
Two measures of reliability have been calculated for the list of objectives and end-terms and its 
subscales: Cronbach's a and Gunman's split-half coefficient Cronbach's a is a reliability measure 
for internal consistency which gives a unique estimate of the reliability for a single usage. It is 
based on the ratio between item variances and the variance of the whole. The split-half method, 
on the other hand, treats one list as if it were two comparable lists (instruments), each of them 
half the size of the original list The reliability of the questionnaire expressed in Cronbach's a is 
0.95; the Guttman split-half coefficient is 0.91. When this questionnaire is subdivided into specific 
learning objectives and end-terms Cronbach's a is 0.93 and 0.85 and the split-half coefficient is 
0.89 and 0.88 respectively. The reliability of the instrument and its subscales is high. In other 
words, the measurements made with the instrument is relatively free of chance. 

In order to determine the extent to which the individual orders of priority correspond to each 
other, the concordance coefficient (Kendall's W) has been calculated (Siegel, 1956; Hays, 1981). 
This coefficient is a measure of the relation among several rankings of N objects or individuals 
(learning objectives) and m subjects (the respondents). It expresses thus the degree of association 
among variables and can vary between the value 0, representing no agreement, and 1 
(representing perfect agreement). From a high value of W one may deduce that the respondents 
have arranged the items on the basis of the same criterion. This does not necessarily mean, 
however, that the order of priority is 'correct', as each respondent may have used an 'incorrect' 
criterion for his/her arrangement. 

The concordance coefficient, W f is defined as: 

W = S 125 

Smax m 2 (N 3 -N) 

in which: 

m S3 number of respondents 

N S3 number of general learning objectives 

S m total number of actual differences in the order of priority of objectives between the 
respondents 

^max m highest possible number of differences in the order of priority objectives between the 
respondents 

Table 4 shows the values of W y as well as the calculated values of A" 2 and the corresponding 
significance levels. 
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Table 4 

Concordance coefficient per respondent group 



Institute 


m 


N° 


W 


X 2 


sign. 


DUE 


15 


6 


0.37 


2152 


< 0.001 


UE 


22 


6 


0.08 


8.94 




OuN 


12 


6 


0.46 


27.48 


< 0.001 



a The reason that N is 6 and not 8 was mentioned in paragraph 2.1 and is discussed again in paragraph 5.1 
^ n.s. = not significant 

From this we may conclude that the DUE respondents are highly consistent in their view on the 
use of practicals in the undergraduate part of educational programmes. In fact, they are 
(statistically) as consistent as the members of the faculty of Natural Sciences at the OuN, for 
which a highly significant degree of concordance was found. For the OuN, such a high value is not 
surprising, considering the fact thr respondents concerned all belong to the same institute and are 
working together as a close-knit group to give shape to innovative education at the OuN. For 
DUE, different institutes spread throughout the world, it is surprising. 



5 Results 

In this chapter we will first discuss the results of the ranking of the general objectives. 
Subsequently the results of the evaluation of the specific learning objectives will be discussed and 
finally the end-terms. 

5.1 General learning objectives 

Before starting with the systematic discussion of the results, attention should be paid to the 
following. On the basis of the results of earlier research into learning objectives for practicals at 
the OuN (Meester et al, 1989), it was decided to discard two of the eight original general 
objectives. After analysis and rearrangement of the 64 specific learning objectives, it was found 
that the two general objectives 4 to formulate hypotheses' and *to remember the central idea of an 
experiment over a significantly long period of time* only corresponded to two and one 
specifications respectively. These specifications could be and were brought under other general 
objectives. The two specific objectives concerning the formulation of hypotheses were brought (as 
component skills) under the general objective Ho solve problems*; the specification of the other 
general objective under 'to clearly describe experiments*. Although these two original general 
objectives may be interesting for further analysis from an intellectual viewpoint, they are 
uninteresting from a practical point of view. General objectives which cannot be operationalized 
into one or more specific objectives cannot really be considered to be general and are not of any 
great importance to education. These two general objectives were therefore left out of the 
discussion and the analysis of the results. As each of the 28 pairs is independent of the other 
pairs, this decision does not have any consequences for the processing of the results. 
The six remaining general learning objectives are: 
A to solve problems 

B to use knowledge and skills in unfamiliar situations 

C to design (simple) experiments to test hypotheses 

D to use laboratory skills in performing (simple) experiments 

E to interpret experimental data 

F to clearly describe the experiment 
These six general learning objectives can be found in appendix 8, together .with the corresponding 
specific learning objectives. 
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In order to gain insight into the evaluation by the respondents of the six general learning 
objectives, we tallied the number of times that a certain general objective was given preference 
over another objective, for the entire research population. According to Thurstone (Swanborn, 
1982) this is the first step towards the analysis of the response on a comparison in pairs. It allows 
one to get a first impression of the order of priority. Figure 2 shows the results. 

general objective 



solve problems (A) 
use knowl/ skills IB) 
design exp (C) 
use lab skills ID) 
Interpret data (E) 
describe exp \*F) 




0 10 SO 30 40 SO 60 

absolute frequency 



Figure 2 

Frequency distribution (max = 75) of the general learning objectives for DUE 



The average scores of the objectives can be obtained by dividing the absolute frequency of each 
objective by the number of respondents. The highest possible score is five, the lowest possible 
scoje zero. In table 5 the general objectives are arranged in order of a decreasing average score, 
i.e. of decreasing perceived importance. 



Table 5 

Ranking of general learning objectives in order of decreasing importance. 



rank 


ave. 


general learning objective 




score 




1 


3.60 


to solve problems (A) 


2 


3.27 


to use knowledge and skills in unfamiliar situations (B) 


3 


2.73 


to interpret experimental data (E) 


4 


253 


to design (simple) experiments to test hypotheses (C) 


5 


2.27 


to use laboratory skills in performing (simple) experiments (D) 


6 


0.60 


to clearly describe the experiment (F) 



The order of priority with regard to the general objectives as expressed by the respondents from 
DUE institutes is quite outspoken (table 5). The difference between the most and least important 
objective is 3.00. 

The most highly rated general objective *to solve problems' (4) scored significantly higher than 
the three least important general objectives C, £>, and F /><0.05; f=2.23, /?<0.05; f=8.53, 

/xO.001 respectively). All the other objectives (B, £, C, and D) score significantly higher than the 
least important general objective Ho clearly describe the experiment' (F) (f=5.62,p<0.001; *=6.96, 
/><0.001; f=5.85,/><0.001; *=3.03,p<0.01 respectively). 
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Specific learning objectives 

The second instrument, a Likert scale inventory on the specific learning objectives and the 
end-terms, was divided into its component parts for analysis and discussion purposes. Appendix 9 
gives a survey of the results of the specific learning objectives sublist arranged in order of 
decreasing importance for DUE institutes. Besides the average value (r) and standard deviation 
(s.d), a normalized score (Z) has been given for each objective. This score makes it easier to 
compare the results of DUE, OuN and UE with each other. This Z-score is based on the averages 
of the 64 specific objectives and indicates how many standard deviations the score for an objective 
is removed from the total average score (Z = 0). Table 6 gives the distribution of the specific 
objectives over the different categories of importance. The results of the OuN and UE have been 
included here as well (Meester et al, 1990). 

Table 6 

Frequency distribution of 64 specific learning objectives arranged in order of decreasing 
importance 

type of number of la 2 3 4 5 

institute respondents 



a 1 =< indispensable (1.00 * x < 1.80); 2 = important (1.80 * x < 2.60); 3 = neutral (2.60 * x < 3.40); 
4 ~ not really necessary (3.40 * x < 4.20); 5 » superfluous (4.20 * * * 5.00). 

A striking aspect of this table is the large number of objectives classified as important by all the 
respondent groups. Furthermore the categories not really necessary and superfluous are empty for 
DUE and UE. This is not really surprising, given the fact that the list of specific objectives is 
based on literature on objectives for practicals in traditional undergraduate science education 
(Kirschner and Meester, 1988). When we look at the 13 specific objectives belonging to the 
category indispensable (1.00 s x < 1.80) we see that four of the objectives are relevant to the 
actual understanding of the purpose of the experiment and to activities prior to the execution of 
the experiment, four of the objectives deal with the actual execution of the experiment and the 
remaining five objectives relate to the processing of experimental data and a description of the 
experiment after the execution of the experiment In table 7 the successive stages have been 
indicated with the headings 'prior to the experiment', 'during the experiment* and 'after the 
experiment*. 



DUE 

UE 

OuN 



15 
22 
12 



13 (20%) 
23(36%) 
11 (17%) 



37 (58%) 
33 (52%) 
37 (58%) 



14 (22%) 
8 (13%) 
10 (16%) 



6(9%) 
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Table 7 

The 13 indispensable learning objectives of the DUE distributed in accordance with three 
consecutive stages 

specific learning objective? x s.d 
prior to the experiment 

to understand what is to be measured in an experiment 1.40 0.83 

to understand the purpose of an experiment 1.53 0.74 

to make order-of-magnitude calculations and estimates 1.60 0.63 

to recognize hazards so as to take safety precautions 1.73 0.70 

during the experiment 

to conduct experiments safely 1.57 0.65 

to observe phenomena in a quantitative way 1.60 0.51 

to use practical (as opposed to theoretical) lab skills 1.67 0.62 

to collect experimental data 1.73 0.80 



after the experiment 






to analyze experiments data to draw conclusions 


133 


0.49 


to interpret reliability and meaning of results 


1.40 


0.51 


to ?pply principles instead of rote formulae 


1.60 


0.51 


to communicate experimental findings in written form 


1.67 


0.82 


to evaluate experimental outcome with respect to a hypothesis 


1.67 


0.90 



The complete text of the objectives can be found in appendix 8 



In conclusion we can say that these indispensable learning objectives belong to the very basic 
skills of doing practical work. Next, we will look at the other end of the list of specific objectives 
(appendix 9). In the neutral category are 14 objectives, which all score relatively high in this 
category (almost all have* < 3.0, see table 8). 



Table 8 

The 14 neutral learning objectives of the DUE 



specific learning objective 


X 


s.d 


general 
objective 


to calibrate instruments 


3.07 


1.28 


D 


to solve difficult scientific problems 


3.07 


1.10 


A 


to incorporate unexpected experimental results in a new model 


3.00 


1.13 


E 


to construct models based on experimental, findings 


2.93 


0.96 


B 


to develop measurement techniques 


2.87 


1.19 


C 


to confirm already known facts and laws 


2.87 


1.13 


E 


to construct models which fit experimental findings 


2.87 


1.13 


B 


to set up lab equipment quickly and correctly 


280 


1.08 


D 


to handle modern equipment 


2.80 


1.08 


D 


to confirm facts, principles and theory from lectures/books 


2.73 


1.22 


E 


to keep a day-to-day lab diary 


2.73 


1.22 


D 


to suggest follow-up investigations 


2.73 


1.03 


F 


to estimate outcome of experimental measurements within 








given precision 


2.67 


1.18 


E 


to translates conceptional definition into set of measurement 








procedures 


2.60 


0.83 


C 



Four of them are specifications of general objective D and obviously do not manifest themselves 
as really important goals for distance education. Two specific objectives are relating to confirming 
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already known facts, laws and theory. Verification of the already known will not be a really 
important goal of modern practicals, a result we also found at the OuN and UE. The remaining 
eight objectives in the category neutral probably aim too high for undergraduate practical. They 
all have to do with higher academic skills. 



End-terms 

Appendix 10 gives a survey of the 38 specific end-terms in order of (decreasing) importance as 
scored by DUE. Here too, the average value (x), the standard deviation (s.d) and the normalized 
score (Z) are given. 

Table 9 gives the distribution of the end-terms over the different categories of importance. The 
results of OuN and UE have again been included here (Meester et al, 1990). 

Table 9 

Frequency distribution of 38 end-terms arranged in order of decreasing importance 

type of number of l a 2 3 4 5 

institute respondents 

DUE 15 4 (10%) 19 (50%) 14 (37%) 1 (3%) 

UE 22 3(8%) 23(60%) 12(32%) 

OuN 12 6(18%) 18(47%) 13(34%) 1(3%) 

a 1 * indispensable (1.00 * x < 1.80); 2 « important (1.80 * x < 2.60); 3 « neutral (2.60 *x< 3.40); 
4 » not really necessary (3.40 * x < 4.20); 5 « superfluous (4.20 * * * 5.00) 

The different institutes evaluate the majority of the end-terms as indispensable or important, a 
relatively large part is rated neutral. None of the end-terms is considered superfluous. 

In DUE, four of the 38 end-terms are assessed as indispensable (see table 10). All four relate to 
higher academic skills, i.e. to attitudes concerning the way scientific problems should be 
approached. 

Table 10 

End-terms considered indispensable by DUE 



specific end-term x 

to approach observed phenomena from a scientific point of view 1.53 

to have a critical attitude to experimental results 1.53 

to solve problems in a critical, academic way 1.71 

to approach a problem with an open mind 1.73 



Close examination of the end-terms rated neutral (14) and not really necessary (1) reveals that 
these end-terms can be grouped around a uore or less common denominator: end-terms with a 
rather 'romantic' or idealistic' view on undergraduate practicals (i.e. experience kinship with the 
scientist), end-terms concerning working attitude and working conditions (i.e. tackle a problem 
with the help of others), and end-terms related to certain academic skills (i.e. determine limits 
under which a theory applies). 
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6 Discussion 

As with the results, the discussion will first focus on the general objectives, then the specific 
objectives and finally the end-terms. 

6. 1 General learning objectives 

From the background data received, we can conclude that those DUE institutes which responded 
in our study can be characterised as monodisciplinary in nature. In this respect they do not differ 
very much from the traditional Dutch universities which responded to an earlier study. Being 
similar in this respect, we will first compare DUE to UE. After doing this we will compare the 
OuN with both DUE and UE. 



6.1.1 General learning objectives: DUE compared with UE 

DUE and UE ranked the general objectives in a very similar manner (see table 11). Four of the 
six objectives were ranker in an identical way by both groups (JJ, Z>, £, and F; ranks 2, 5, 3, and 6 
respectively). Only two objectives differ, namely A and C. Objective^, to solve problems', ranked 
fourth in UE, is for and away the most important objective in DUE. Objective C, *to design 
(simple) experiments to test hypotheses', ranked as most important in UE is ranked fourth in 
DUE. As a matter of fact, objective C was ranked significantly lower in DUE than .4 (r=2.17, 
p<0.05). 

Although the ranking is very similar, the differences between the scores are more pronounced in 
DUE than in UK This can be seen in the number of significant differences between the scores 
from the paired comparison within DUE (7) and those within UE (3). The most pronounced 
difference was the scoring of the general objective F, Ho clearly describe the experiment'. 
Although this objective received the lowest score in both educational types, DUE scored more 
than a full point lower than UE. Clearly, though this general objective is not a priority in either 
type of institution, the importance in DUE is almost nil. 

On the basis of these data, it appears that there are no really noteworthy differences between 
DUE and UE. 



Table 11 

Ranking of importance of general learning objectives 



general learning objectives 


rank** 


ave 


rank 


ave 


rank 


ave 




OuN 


score 


DUE 


score 


UE 


score 


to solve problems (A) 


4 


2.83 


1 


3.60 


4 


255 


to use knowledge and skills in unfamiliar situations (B) 


1 


3.58 


2 


327 


2 


2.73 


to design (simple) experiments to test hypotheses (C) 


3 


3.08 


4 


233 


1 


3.09 


to use lab skills in performing (simple) experiments (D) 


6 


050 


5 


2.27 


5 


2.05 


to interpret experimental data (£) 


2 


3.42 


3 


3.27 


3 


2.68 


to clearly describe the experiment (F) 


5 


1.42 


6 


0.60 


6 


1.77 



1 ~ mott important; 6 « least important 



6.1.2 General learning objectives: OuN compared to DUE and UE 

In addition to comparing DUE with UE, we can also compare the OuN with both of the 
aforementioned. In this way we can see whether or not the OuN is unique in that it is a distance 
university (DUE and OuN should be more similar to each other than either is to UE) or whether 
it is unique in the fact that it offert an interdisciplinary science programme (in which case DUE 
and UE should be more similar to each other than either to to the OuN) or both. 
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Kendall's rank correlation coefficient (r) 2 is a good measure of the correlation between two 
ordinal measures. This correlation coefficient is based on the number of pairs of ranks that are 
ordered in the same direction in two different sample groups and varies from 0 by complete 
disagreement to +1 by complete agreement It is thus, in effect, an enumerator of the extent of 
disagreement in the rankings in the two groups (Glass and Stanley, 1970; Siegel, 1956). In our 
case, those measures are the ranks given by each of the types of institutes to the general 
objectives. This leads to three comparisons, namely DUE-UE, OuN-DUE, and OuN-Ufs. None of 
the comparisons proved to be significant (r = 0.33, 0.47, and 0.60 respectively, p>0.05). From 
this we can conclude that none of the institutes is significantly similar to any of the other two; 
thus the three types are statistically different. A possible explanation for this is that the factors 
which differentiate distance education from face-to-face education and monodisciplinaiy education 
from multi- or interdisciplinary education play a role in this uniqueness. 
A confounding variable which we did not take into account when we set up this research may be 
the cultural (dis)similarities between the respondent groups. By this we mean the educational 
climate, culture and history of the different respondent groups. Although the OuN and UE differ 
along the two dimensions discussed thus far, they do have a certain similarity that cannot be 
discounted All of the individual respondents come from a West European (Dutch) scientific and 
academic culture. Although they work in strikingly different educational institutes, they have 
similar, if not for all intents and purposes identical, educational histories. They all attended one 
or more of the 9 traditional Dutch universities with Natural Science curricula, attended the same 
types of secondary schools, and had to meet very similar requirements to achieve the positions 
which they fulfil If we look at the values for r, we see that although none of the values is 
significant, they increase from DUE-UE (where the nature of the institution and the cultural 
factors are dissimilar) through OuN-DUE (where the nature of the institution is similar but the 
cultural factors are dissimilar) to OuN-UE (where the nature of the institution differs, but the 
cultural factors are similar). Table 11 shows the average scores for each of the general objectives 
for each of the institutional types. 

The similarity gradient in Kendall's r discussed in the preceding paragraph can also be seen in a 
comparison of the average scores for each of the groups of respondents for the general objectives. 
UE and DUE differ significantly with respect to their scoring of objectives A and F (Ax=1.05, 
f=2.12,p<0.05 and Ax=1.17, f=3.30,/><0,005 respectively). The ability to solve pro >lems (A) is 
more highly valued by DUE in relative terms (rank 1 versus rank 4), but also in absolute terms. 
The same is true for being able to describe an experiment (F). 

The OuN and DUE differ significantly with respect to their scoring of objectives D and F 
(Ajc=1.77, f=3.59, p<0.005 and Ax=0.82, /=2.31,p<0.05 respectively). DUE values the use of lab 
skills to perform (simple) experiments (D) more highly than the OuN, not only relatively (rank 4 
versus rank 6), but also in absolute terms. The opposite is the case for being able to describe an 
experiment (f). Here it is the OuN which places more value on this objective. This is well in line 
with the general philosophy of the OuN where the end-terms for the Natural Sciences stress 
written and oral communication while tempering the necessity for working in a laboratory. This 
difference with other distance universities may thus be the result of the Dutch Open university's 
strong resolve to interdisciplinary programming on the one hand and its steadfast belief in the 
freedom for students to study where and when they wish. 

The results obtained with respect to objective F need to be discussed further. The low score, for 
all institutes, of general objective F 'to clearly describe the experiment' is quite surprising, because 
the publication of research results in scientific magazines and the presentation of results at 
congresses and conferences is quiet common in scientific circles. This result is even more 
surprising if one considers the high score of the specific objectives which are part of this general 



^ Other possible measures are Spearman's rank correlation coefficient which it based upon the absolute differences in the 
rankings by two groups and Kendall's coefficient of concordance which was described in section 4.2. We calculated all three, 
and they all yielded the same conclusions. We chose to present (t) here. 
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objective and in particular those specifications relating to written communicative skills (see 
paragraph 5.2). This discrepancy may be the result of the use of the verb 'to describe'. According 
to the various taxonomies of objectives (Bloom et al., 1956; Gronlund, 1970; Klopfer, 1971) 'to 
describe' can be interpreted in two ways. According to Bloom the verb first of all fi \ in the lowest 
category of objectives within the cognitive domain namely the category of knowledge objectives. 
Characteristic verbs are: to reproduce, to remember, to recognize, to describe, to enumerate, etc 
The emphasis is on remembering or spontaneously recognizing subject matter, facts, definitions, 
connections and methods. 

Apart from this interpretation as a low* cognitive objective, it is also possible to interpret the 
verb as a relatively 'high' cognitive objective. In this respect the word to describe is used in the 
sense of synthesizing. Synthesis is the ability to put parts together to form a new whole. This may 
involve the production of a unique communication (theme or speech), a plan of operations 
(research proposal), or a set of abstract relations (scheme for classifying information)' (Bloom, 
1956). Here, emphasis is put on creative behaviour, the creation of new patterns or structures. 
According to Gronlund (1970) objectives such as the ability to develop a proper line of reasoning, 
the ability to give a good lecture or the ability to summarize the essentials of an experiment 
belong to the category synthesis. Apparently the respondents, like the respondents of the OuN, 
have interpreted the general objective in the sense of low cognitive, especially in comparison with 
the other general objectives. By contrast, the specific objectives relating to communicative skills 
are interpreted as higher, more creative and therefore both UE institutes and DUE institutes 
count them among the indispensable objectives. 

In a comparison of average scores for the OuN and UE, the only significant difference in average 
score is on objective D (At =1.55, *=3.80,/?<0.001). Though the relative difference between the 
groups with respect to the use of laboratory skills is small (rank 5 for UE and rank 6 for OuN), 
the absolute difference is very large. Again we see here the OuN's dedication to approach the 
Natural Sciences in a way tnat is different from more traditional, monodisciplinary, universities. 
The OuN focuses on and strives to provide "a type of education which enables its graduates to 
learu to think in a problem-oriented manner" (End-terms, 1986). In other words, what the OuN 
graduate misses in laboratory skills is/must be more than compensated by training in problem 
oriented and problem solving thinking. 

Finally, the lack of significant differences between the groups can be seen in the comparison of 
the absolute differences between the different response groups (see table 12). Metaphorically 
speaking, the OuN and UE are in essence children of the same parents (almost twins if you wish), 
who spent their formative years in similar environments and have only recently been split up and 
placed, in vastly different environments. Both nature and nurture are thus similar. Is it strange 
then that although differing greatly on certain items, their overall behaviour is very similar? 

Table 12 

The difference in the averages (Ax) for the general learning objectives when comparing the 
various institutes. Values with a A > 1 or * 1 are printed in bold letters. 



general objective 


OuN-DUE 


OuN-UE 


DUE-UE 


to solve problems (A) 


4.77 


0.28 


1.05 


to use knowledge and stalls in unfamiliar situations (B) 


031 


0.85 


0.54 


to design experiments to test hypotheses (C) 


055 


-O.Gi 


•0.56 


to use laboratory skills in performing experiments (D) 


-1.77 


-155 


0.2 


to interpret experimental data (E) 


0.15 


0.74 


0.59 


to clearly describe the experiment (F) 


0.82 


-0.35 


zU2 


sum of the absolute differences: l\Ax\ 


4.37 


3.78 


4.13 
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6.2 Specific learning objective* 

Drawing conclusions from the results of the specific learning objectives requires special care. 
There is a certain degree of subjectivity with regard to the interpretation of each specific 
objective. This study can only indicate a tendency in objectives which are considered more or less 
important as well as indicate the corresponding general objiectives. It is almost impossible to 
achieve a one-to-one correspondence, because certain specific learning objectives might 
occasionally be considered to belong to different general objectives. The distinction between 
specific learning objectives and end-terms is also not always unequivocal. We consider to solve 
difficult scientific problems* for instance to be an objective because we interpret it as the ability to 
solve problems, partly prior to an experiment However this could also be considered as an 
end-term of a study of the Natural Sciences. Despite the above mentioned interpretation problems 
we are of the opinion that it is safe to analyze the results of the specific objectives and to draw 
conclusions from this analysis. 



6.2.1 Specific learning objectives versus general learning objectives 

The subsumption of each of the specific objectives to one of the six general objectives was 
presented in a previous article. This classification is, of course, open to discussion. It is, thus, 
interesting to use the results of this research to check as to whether or not there is any empirical 
basis for the classification. 

One way to do this is by calculating Cronbach's <r, a measure of reliability, for each of the general 
objectives. Cronbach's a is a measure of homogeneity between the variance of the separate 
objectives and the variance of the scale (each general objective) as a whole (see paragraph 4.2). 
Table 13 shows the value of a for each of the general objectives. In calculating the scores, we 
used the responses from the OuN, UE, and HVE together to achieve a maximum sample size. 
This table gives a strong indication that the specifications of the general learning objectives as 
defined in Kirschner & Meester (1988) and revised in Meester et al (1990) are fairly homogenous 
and that the classification is very well plausible. Only the objective to solve problems' (A) has a 
rather low a (0.58). 

Table 13 

Cronbach's a for the specifications of the general learning objectives 



general learning objective number of Cronbach's 

specifications a 

A -to solve problems 10 0.58 

B -to use knowledge and skills in unfamiliar situations 6 0.77 

C -to design experiments to test hypotheses 8 0.78 

D -to use laboratory skills in performing experiments 17 0.91 

E -to interpret experimental data 16 0.75 

F -to clearly describe the experiment 7 0.73 



In order to determine whether a relationship exists between perceived importance of the general 
objectives and the ratings of the constituent specific objectives we calculated an average Z-score 
for each group of specifications, where Z is the normalized score for a objective. We arrived at 
this score by summing up the Z-scores for all of the specific objectives belonging to a general 
objective and dividing this by the number of specifications/The average Z-scores for the different 
group(ing)s of respondents are given in table 14. 
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Table 14 

Average Z-scores for the specifications per general learning objective and the differences between 
them for the different response groups 



general objective 


OuN° 


DUE 11 


UE? OuN-DUE? OuN-UE* 


DUE-VE* 


to solve problems (A) 


+039 


+0.02 


-0.15 


+037 


+054 


+0.17* 


to use knowledge and skills 














in unfamiliar situations (B) 


+0.42 


-0.40 


-0.43 


+0.82 


+0.85 


+0.03 


to design experiments 














to test hypotheses (C) 


-0.21 


-0.11 


-0.18 


-0.11 


-0.03 


+0.07 


to use laboratory skills 














in performing experiments (D) 


-0.85 


-0.05 


+0.07 


-0.80 


-0.92 


-0.12 


to interpret experimental 














data (£) 


+0.43 


+024 


+0.12 


+0.19 


+031 


+0.12 


to clearly describe the ^ 














experiment (F) 


+0.40 


+0.09 


+036 


+0.31 


+0.04 


-0.27 



a A positive score indicates above average evaluation of the objective; a negative score indicates below average 
evaluation. 

* A positive difference indicates a greater importance attached to the objective by the first institute; a negative 
difference indicates the opposite. 

It is evident that the largest differences between the OuN on the one hand and DUE and UE on 
the other are for the general objectives B, D and (to a lesser degree) A. As stated earlier, the 
OuN finds the use and development of laboratory skills (£>) much less important than does DUE 
and UE (difference is negative), but deems objectives dealing with the use of knowledge and skills 
in unfamiliar situations (B) and solving problems (A) as being more important than the other 
institutions (difference is positive). Not strange when one takes into account the objectives of the 
Natural Science programme of the OuN. The differences between DUE and UE are only very 
modest We can prioritize the general objectives based upon the scores in table 13 and compare 
this to the rankings obtained by the paired comparison. This has been done in table 15. 

Table 15 

Rough sequence of general learning objectives 3 determined on the basis of average Z-scores for 
the specific learning objectives and the paired comparison. 



type of 


order 


1 


2 


3 


4 


5 


6 


institute 


based on 














OuN 


Z-score 


0.43 


0.42 


0.40 


0.39 


-0.21 


-0.85 




spec, objectives 


E 


B 


F 


A 


C 


D 




paired 
















comparison 


B 


E 


C 


A 


F 


D 




average score 


3.58 


3.42 


3.08 


2.83 


1.42 


030 


DUE 


Z-score 


0.24 


0.09 


0.02 


-0.05 


-0.11 


-0.40 




spec, objectives 


E 


F 


A 


D 


C 


B 




paired 
















comparison 


A 


B 


E 


C 


D 


F 




average score 


3.60 


3.27 


2.73 


Z53 


2.27 


0.60 


UE 


Z-score 


036 


0.12 


0.07 


-0.15 


-0.18 


-0.43 




spec, objectives 


F 


E 


D 


A 


C 


B 




paired 
















comparison 


C 


B 


E 


A 


D 


F 




average score 


3.09 


2.73 


2.68 


235 


2.05 


1.77 



a 1 « most important; 6 « least important 
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The table shows a fairly large degree of similarity between the two ways of ranking the general 
objectives for the OuN. It is primarily objective F to clearly describe the experiment 1 and C to 
design experiments to test (simple) hypotheses 1 , which yield the greatest discrepancies* For 
objective F we have already discussed the role which interpretation of the verb 'to describe 1 may 
have played in the paired comparison in earlier reports and articles (Kirschner et al, 1991; 
Meester et al, 1990). This objective scored lower on the paired comparison than on the average 
Z-score for all three groups. If we give F a higher priority more in agreement with the order 
based on average Z-scoies, then the order of A and C will only be reversed for the two 
rankings.The two ways of ranking for DUE (and UE) are very different A remark can be made 
about the difference in ranking for objective B 'to use knowledge and skills in unfamiliar 
situations' in DUE and UE. While this objective ranks rather high based upon the paired 
comparison, it receives the lowest rank based upon the specifications. This may be due to the fact 
that unfamiliar situations (generally speaking and thus in the ranking of the general objectives) 
should occur in UE and DUE. In practice, though, these situations occur sparsely in 
undergraduate practicals (specifically speaking). This is probably reflected in the scoring of the 
specific objectives. 

The ranking based on the average Z-scores for DUE and UE are very comparable; for the ranking 
based on the paired comparison general objectives A and C have changed the positions 1 and 4. 
This is strangely in the light of the practically same position in the ranking of average Z-scores. 



6.2.2 Specific learning objectives: DUE compared UE 

To compare the various institutes we make use of Z-scores. Appendix 11 gives the differences 
between DUE and UE in order of the decreasing absolute differences in Z-scores ( | AZ | ). In 
order to assess the Z-scores we have to determine which difference in Z-scores is sufficiently large 
to characterise a difference in the identity of the institute and at the same time which disparity in 
Z-scores is small enough to ensure that the evaluation of a specific objective is virtually the same 
for both institutes. For the upper limit we have chosen for a disparity of one standard deviation; 
for the lower limit for a disparity of half a standard deviation. According to these criteria DUE 
differs from UE in only eight specific objectives. For these objectives either DUE puts a great 
deal of emphasis on them and UE does not or vice versa. In table 16 these objectives are shown 
in order of their decreasing AZ scores. 

Table 16 

Specific learning objectives in which DUE differs greatly from UE 



specific learning objective AZ (DUE - UE) general 

objective 

to derive and evaluate relationships 1.25 A 

to confirm facts, principles and theory from lectures /books 1.24 E 

to design relevant observation techniques 1.22 C 

to confirm already known facts and laws 1.13 E 

to apply principles instead of rote formulae 1.12 E 

to estimate outcome of experimental measurements within 

given precision -1.21 E 

to suggest follow-up investigations -1.05 F 

to handle modern equipment -1.01 D 



Two of the objectives which DUE considers to be much more important than UE refer to the 
confirmation of established facts, laws and theories. However, also DUE places them in the 
category neutral (see section 5.2), so these objectives are not really important 
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A more striking result is that DUE and UE do not markedly differ with respect to 37 objectives 
(see appendix 11 and table 17). It would also be possible to draw this conclusion by comparing 
the indispensable objectives for DUE and UE respectively as they have been classified according 
to the three stages of the execution of an experiment: 'prior to the experiment', 'during the 
experiment* and 'after the experiment' (for DUE see table 8 and for UE see Meester et al., 1990). 
The learning objectives in the stages 'prior to experiment' and 'during the experiment' appeared 
to coincide almost completely, except for three additional objectives for UE dealing with the 
properly planning of an experiment, the understanding of measurement of different phenomena 
and the carrying out of accurate measurements. For UE the objectives categorized in 'after the 
experiment' contain five of the six objectives of DUE (except to apply principles instead of rote 
formulae) and an additional eight objectives, which deal with presenting the results of a 
experiment (3), evaluating experimental results (3), processing experimental data (1) and applying 
statistics (1). If we look up these specific objectives in the list of mean values for DUE 
(appendix 8) they all appear to have a mean value of * 2.0. So there is only a modest difference 
between the evaluation of DUE and UE respondents. In conclusion we can say that the 
differences between DUE and UE with regard to the appreciation of specific objectives for 
undergraduate practical are very small. 



6.2.3 Specific learning objectives: OuN compared to DUE and UE 

In addition to comparing DUE with UE, we can also compare the OuN with both DUE and UE 
and indicate the various discrepancies. In appendix 12 and 13 the differences between OuN and 
DUE and between OuN and UE are given in order of decreasing absolute difference in Z-scores 
(|4Z|). Again we will draw a line at \AZ\ = land|4Z| = 0.5. With the help of these limits 
the specific learning objectives can be classified in three groups as seen in table P where the 
number of objectives per institute combination is given. 

Table 17 

The number of specific learning objectives classified according to the size of | AZ | in a 
comparison of DUE and UE, OuN and DUE and OuN and UE 

/ 

comparison of \AZ\ * 1 0.5 < \AZ\ < 1 \AZ\ * 0.5 

DUE and UE 8 19 37 (appendix 10) 

OuN and DUE 25 16 23 (appendix 11) 

OuN and UE 21 20 23 (appendix 12) 

Some results in this table are noteworthy to indicate: 

- The number of specific learning objectives on which DUE and UE vary a great deal is not very 
large (8); the number of objectives they practically agree on is considerably larger (37) (see also 
section 6.2.2). 

- The number of learning objectives showing dissimilarities between the OuN and DUE or UE is 
almost equal (although the type of objective concerned may vary a great deal) and this 
particular number (25 and 21 respectively) is approximately three times as high as the number 
of differences between DUE and UE 

- The number of learning objectives on which the OuN and DUE on the one hand and the OuN 
and UE on the other hand hold the same view is the same (namely 23). 

From the above mentioned outcomes one might conclude that as far as practicals are concerned 
the OuN differs much more from DUE and UE than DUE and UE differ from one another! This 
conclusion is more or less in accordance with the specific character of the Natural Sciences at the 
Open university of the Netherlands. The type of science education offered by the OuN is more 
inter- or multidisciplinaiy than education in the DUE and UE institutes, which are all 
monodisciplinaiy. This requires a different attitude with regard to monodisciplinary practicals. We 



ERIC 



29 



23 



PRACTICAL OBJECTIVES 



can look at the specific learning objectives with considerable differences in Z-scores in another 
way. We have seen (table 17) that the OuN varies to a large extent from DUE with regard to 25 
specific objectives. We can check how many of those objectives also occur in the list of 21 
objectives the OuN differs strongly from UK This proves to be the case with 13 of them. These 
13 objectives could be considered as the objectives which provide specific information about the 
OuN; objectives which clearly distinguish the OuN from DUE and UE. These may be objectives 
which the OuN considers much more important than DUE and UE (AZ * 1) or objectives which 
OuN values as much less important than DUE or UE (AZ <z -1). It does not necessarily mean 
that DUE or UE does not value these objectives as important, but merely that the OuN values 
them as either much more important or much less important (see table 18). 

Table 18 

Specific learning objectives with a large difference between the OuN and both DUE and UE 
specific learning objective 

to incorporate unexpected experimental results in new model 
to solve difficult scientific problems 
to solve problems in a multi-solution situation 
to decompose larger problems into smaller problems 
to derive testable hypotheses from theories 
to be flexible in modifying experiments 
to react adequately to unforseen results 

to use practical (as opposed to theoretical) lab skills 
to collect experimental data 
to put basic laboratory techniques to use 
to manipulate apparati 
to carry out accurate measurements 
to conduct experiments safely 



(Ou-DUE) AZ(Ou-UE) 


general 






objective 


239 


214 


E 


137 


137 


A 


1.24 


1.57 


A 


1.20 


1.04 


A 


1.20 


134 


A 


1.08 


1.12 


D 


1.08 


1.09 


B 


-208 


-1.86 


D 


-1.80 


-1.72 


D 


-1.61 


-1.47 


D 


-1.55 


-1.46 


D 


-1.24 


-1.70 


D 


-1.23 


-1.47 


D 



The learning objectives in the list with negative AZ values all relate to experimental laboratory 
skills during the actual realization of the experiment. All of them are specifications of general 
objective D 'to use laboratory skills in performing (simple) experiments'. Table 8 shows that DUE 
values three of these practical laboratory skills as indispensable objectives and UE four of them, 
but for an OuN graduate in science they are not really necessary (Meester et al, 1989). 
Four of the learning objectives with a AZ z +1 are specifications of general objective^ 'to solve 
problems'. The faculty of Natural Sciences at the OuN shows a clear preference for the 
achievement of the problem-solving skill. Since the amount of time and money that can be spent 
on practicals is not limitless, the preference for this higher academic skill must be gone at the 
cost to lower level manual skills dealing with the use of laboratory skills. The other three specific 
objectives with AZ * +1 are related in some way with reacting to unforseen or unexpected 
experimental results. This flexibility, which in its ultimate form can be seen as the ability to 
incorporate serendipity into the scientific process (the classic example being the discovery of 
penicillin by Fleming), can be seen as a benchmark academic skill in al sciences. The ability to 
change one's view in light of new empirical evidence and not get caught up in existing scientific 
dogmas is the essence of the true scientist. Paul Dehart Hurd (1969) expressed this in the 
following way: "Since the natural sciences are distinguished by a continual flow of new knowledge, 
by refinements of existing knowledge, accompanied by new theories and knowledge, the teaching 
of science must reflect the same dynamic characteristics" (p. 9). 
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63 End-terms 

In this paragraph we compare the results of end-term evaluations for institutions of higher 
distance education (OuN and DUE) and university education in the Netherlands (UE). 

A division of the specific end-terms into three groups based on the \AZ\ score (see appendices 
14, IS and 16) generates the following picture when comparing OuN, UE and DUE (table 19). 

Table 19 

The number of specific end-terms classified according to the size of \AZ\ in a comparison of the 
OuN, UE and DUE 



comparison of 

DUE and UE 
OuN and DUE 
OuN and UE 



\AZ\ */ 

4 
10 
6 



0.5 < \AZ\ < 1 

14 
13 
11 



\AZ\ *0.5 

20 
15 
21 



(appendix 14) 
(appendix 15) 
(appendix 16) 



The number of end-terms showing a considerable difference in appreciation (\AZ\ z 1) is 
approximately twice as high in the comparison of OuN and DUE as it is in the other 
comparisons, while the number of end-terms classified similarly ( | AZ | * 0.5) is less. This result 
suggests that the OuN differs more from DUE than it does from UK In fact, the results 
presented in table 17 for the specific learning objectives indicate a similar conclusion (see section 
6.23). In the next sections we will look at the nature of the differences between the institutions. 



6.3.1 End-terms: DUE compared to UE 

When applying, as we did in paragraph 6.2, the criterion that a | AZ \ * 1 indicates an essential 
difference between the institutes to be compared, DUE differs considerably from UE with regard 
to only four end-terms (table 20). 

Table 20 

End-terms showing considerable differences between DUE and UE 



specific end-term AZ(DUE-UE) 0 

to build a framework for facts, principles and theory from lectures/books +2.20 
to discover limitations of a theory/model -1.78 
to work in research and development labs -1.65 
to work in groups to solve scientific problems -1.01 



AZ * +1 = this cnd-tcrm is considered much more important by DUE than by UE 
AZ s -l = this end-term is considered much less important by DUE than by UE 



Two of the end-terms which DUE considers to be much less important than UE refer to working 
conditions (*work in research and development labsTwork in groups to solve scientific problems') 
which are hard to fulfil wiuiin distance education. 

Within DUE much emphasis is put on integration of (theoretical) knowledge ('build a framework 
for facts, principles and theory from lectures/books*) which was also found within OuN (see table 
22). This was also the case with respect to the specific objectives which pertain to the 
confirmation of facts, principles and theories. Clearly DUE sees the practical as a vehicle for the 
conveyance/confirmation of theoretical knowledge. 
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6.3.2 End-terms: OuN compared to DUE and UE 

Tables 21 and 22 give a survey of the essential differences in end-term evaluations between OuN 
and DUE and OuN and UE respectively. 

Table 21 

End-terms showing considerable differences between OuN and DUE 



specific end-term AZ (OuN - DUE) 0 

to discover limitations of a theory/model +1.43 

to work independently of others +1.28 

to formulate a problem that can be researched + 1.16 

to work in groups to solve scientific problems +1.13 

to survey literature relevant to some problem +1.02 

to do experiments -1.85 

to use mental skills inherent to professionals -1.27 

to illustrate facts, principles and theory of lectures/books -1.14 

to use motor skills inherent to professionals -1.01 

to use the lab as an instrument for discovery -1.01 



AZ * +1 = this end -term is considered much more important by OuN than by DUE; 
AZ i -1 = this end-term is considered much less important by OuN than by DUE 



Table 22 

End-terms showing considerable differences between OuN and UE 



specific end-term (OuN - UEf 

to be self-confident and independent +152 
to build a framework for facts, principles and theory from lectures/book + 1.49 

to survey literature relevant to some problem +1.12 

to do experiments -2.19 

to work in research and development labs -153 

to use the lab as an instrument for discovery -131 



a AZ * +1 = this end-term is considered much more important by OuN than by UE 
AZ * -1 = this end-term is considered much less important by OuN than by UE 

From comparison of tables 19, 20 and 21 the following conclusions may be drawn: 

- Three end-terms showing differences between OuN and DUE also turn up in the comparison 
between OuN and UE (tables 20 and 21). Two of those relate to the execution of experiments 
('do experiments' and 'use the lab as an instrument for discovery*). Such end-terms are not 
prominent within the OuN-programme for distance education in the Natural Sciences. The 
end-term 4 to survey literature relevant to some problem' is more important to OuN than it is to 
DUE and UE. However, it should be noticed that this end-term is also rated important within 
the latter institutes. 

- The end-term to build a framework for facts, principles and theory from lectures/books* is more 
important to DUE and OuN than it is to UE (tables 19 and 21). 

- Although there is considerable difference in the appreciation of the end-term 'to formulate a 
problem that can be researched* between OuN and DUE (AZ = +1.16), this end-term was 
evaluated as important in DUE. For the end-term to use motors lolls inherent to professionals' 
a AZ (OuN - DUE) of - 1.01 was found. Since this end-term was rated neutral within DUE, this 
end-term is not very important to both institutes (table 20). 
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The end-term to work in groups to solve scientific problems* is valued higher within OuN and 
UE than within DUE (tables 19 and 20). 



Conclusions 

In light of the results presented in this report we can make a number of conclusions with respect 
to learning objectives and end-terms for undergraduate practical^ and for the differences and 
similarities between the OuN, UE, and DUE. 

With respect to DUE we can conclude that: 

- The different institutes making up the sample population for DUE approach piacticals in the 
Natural Sciences in a similar way. This can be seen on the coefficient of concordance (Kendall's 
W). The degree of concordance across the sample population is large. As a matter of fact it's 
level of significance is about the same as that of the OuN. This is rather remarkable considering 
the differences between the individual respondents with respect to culture, language, educational 
background, etcetera. The only two things which they all appear to have in common is that they 
all are institutes for distance education and they all approach the Natural Sciences in a 
monodisciplinary way. 

- The ability to solve problems, interpret experimental data, and use knowledge and skills in 
unfamiliar situations are far and away the most important general objectives for the students 
attending these institutions to achieve. The importance of the first two are also reflected in the 
ratings of their component specific objectives. The operationalisations of the third general 
objective are, on the other hand, ranked very low. 

- The ability to describe an experiment is far and away the least important general objective for 
the students attending these institutions to achieve. When the component specific objectives 
subsumed under the general objectives are used to calculate the relative importance of the 
general objectives, then the ability to describe an experiment jumps from last to second place. 

- The specific objectives which DUE rates as indispensable are primarily elementary laboratory 
skills relating to an experiment. Those objectives which were rated as neutral are, on the other 
hand, primarily higher academic skills relating to reasoning and problem solving. 

- None of the 64 specific objectives is considered not really necessary or superfluous. 

- Four of the 38 end-terms are valued as indispensable. They relate to attitudes concerning the 
way scientific problems should be approached 

In analyzing the reactions of the respondents for the general learning objectives, the specific 
learning objectives and the end-terms we note the following differences between the three groups: 

- The difference between the OuN and DUE is larger than the difference between the DUE and 
UE. This is probably the result of the monodisciplinarity of the programmes in DUE and UE as 
opposed to the OuN which strives towards an interdisciplinary programme. Apparently, the type 
of programme weighs more heavily on the way in which the faculty members think than the 
nature of the institution. 

- Each of the three types of institution is statistically unique from the other two. Seeing as how 
each of the groups represents one of the cells in the matrix presented in section 1, this was to 
be expected. 

- A strange artefact showing up in the results is the gradient of similarity with respect to the 
rating of the general objectives made visible through the calculation of KendalTs r. One would 
expect OuN and DUE, which differ on one of the two dimensions (both are institutions for 
distance education; the type of programme offered is different) or DUE and UE, which also 
differ on one dimension (the nature of the institution is different; both offer monodisciplinary 
programmes) to be more similar than OuN and UE, which differ on both dimensions. The 
results show that DUE-UE are least similar, followed by DUE-OuN, with OuN-UE being most 
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similar. Apparently there is another factor (possibly a cultural one) which is stronger than both 
of the dimensions which causes this anomaly. 

- There is a large degree of similarity between UE and DUE with respect to the specific 
objectives. The specific role which we thought that the nature of the institution would play in the 
evaluation of the objectives by the respondents did not crystallise here. 

- The OuN differed from UE and DUE with respect to the specific objectives. This points in the 
direction of the expected role that the type of programme offered would play in the evaluation of 
the objectives by the respondents. 

- The OuN distinguishes itself from UE and DUE primarily in the much lower evaluation of 
"real" practical skills (laboratory performance skills - specifications of general objective D) and a 
higher evaluation of problem solving skills. 
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Appendix 1 Order of priority of specific learning obj«ctiv#e from 
research within tha OuM 

The first column gives the Item nunber for the specific learning objective from the Inventory 
(appendix 7). The letters 1n the second column correspond with the general learning objectives* 







Objective 


x 


z 


s.d. 


g 


E 


Interpret reliability and meaning of results 


1.25 


1.64 


0.45 


16 


E 


Assess relevance of exD. data with reaard to hvoothesis 


1*50 


1.24 


0.52 


34 


E 


AddIv elementary notions of statistics 


1*50 


1.24 


0.67 


23 


A 


Derive testable hypotheses from theories 


1.67 


0.98 


0.49 


71 


F 


Describe central asoects of an experiment 


1.67 


0.98 


0.49 


97 


E 


Relate exp. outcomes to a particular theory 


1.67 


0.98 


0.49 


102 


E 


Fvaluatp diff. exnected & actual results 

LIUI UdlC Will* vnyvVb&U *» QWiUB 1 ■ 1 b« 


1.67 


0.98 


0.49 


21 


A 


Decomoose larae to smaller oroblems 


1.67 


0.98 


0.65 


91 


A 


Undprstand what is to be measurpd in an exD. 


1.67 


0.98 


0.65 


1 


E 


Makp nrdpr-nf ~mafmi tiide calculations and pstimatps 


1.75 


0.85 


0.75 


OA 


A 

n 


llndor^cf anH thp nnrnocp an pvnprimont 
uiiuci 9 Lai i vi cue uui |iusc vi an cawci iiijcmi 


1 75 

X . / w 


0.85 


0.87 


U*T 


c 


Rprnnn hp7Pt*HQ so ac to take safetv nrerautions 
ixccuiju* iia&ai us su as iu kikc aaicLjr pi c^auLiuiis 




0.73 


0 39 


47 


L. 


Fv/alnatp pyn niitrfwnp with resnert to a hvnnthesis 


1.83 


C.73 


0.58 




D 

D 


Annlv/ known nyinrinleQ tn npw Qi+natinnQ 
n|iuijf Mii/wn ui i iici pica iu new a i iuai luiid 


1.83 


0.73 


0.58 


87 


E 


Analvze exD. data to draw conclusions 


1.83 


0.73 


0.58 


94 


8 


React adequately to unforeseen results 


1.83 


0.73 


0.72 


5 


D 


Observe phenomena in a qualitative way 


1.83 


0.73 


0.83 


17 


F 


Present essentials of an exp. in written form 


1.83 


0*73 


0.94 


22 


c 


Design subsequent exp* Involving phenomena 


1.92 


0*58 


0.79 


43 


E 


Incorporate unexpected exp* results in new model 


1.92 


0*58 


0.79 


25 


F 


Communicate exp* findings in written form 


1.92 


0*58 


0.90 


44 


D 


Be flexible in modifying exp* 


1.92 


0*58 


0.90 


54 


A 


Derive & evaluate relationships 


1.92 


0*58 


0.90 


41 


B 


Recognize & define scientific problems 


2.00 


0*46 


0.43 


26 


E 


Use obtained data to make estimates in new situations 


2.00 


0*46 


0.60 


33 


A 


Use exp* data to solve specific problems 


2.00 


0*46 


0.60 


63 


A 


Solve problems in a multi -solution situation 


2.00 


0*46 


0.60 


77 


c 


Properly plan an experiment 


2.00 


0*46 


0.60 


31 


c 


Design an exp* to verify a theory /hypothesis 


2.00 


0*46 


0.74 


10 


E 


Fstimate outcomp of pxd. meas. within Given nrecision 


2.00 


0.46 


0.95 


35 


B 


AddIv current knowledoe in sol vino new oroblems 

rif v IT vilJ 1 will ^§IVf? 1 Wwlv ill dvl T 1 iiu n\*n V * w * vnit^ 


2.00 


0.46 


1.13 




c 


Fvaluatp rnnt ri hut i nn direct to deriupd prrors 
Lvai uaic cuiiLi luuiiun ui i cvi iu vici i vcu ci i ui s 


2.08 


0*33 


0.51 


55 


F 


Sumnariyp an pxn» haspd on rpsults 

tJvalalKII 1 UI 1 CAp* UQ«CU VII 1 C3U 1 L J 


2.08 


0.33 


0.67 


Rfi 


r 


Understand scone A limits of pyn techninups used 

UilMCI SLailU 9vU|n> Ul 1 Hill t9 UI CA|J» CCCIIIII^UCS U9CU 


2*08 


0.33 


1.00 


42 


F 


ronmtn'irjttp pxn. findinas in oral form 

vMiiiRiii i vU lc ca yj • i i nv4 1 1 iy j i ii vi at i vi m 


2*08 


0.33 


1.08 


68 


D 


Handlp wastp safelv 


2.17 


0.19 


0.94 


45 




Construct modpls based on exn. findlnns 


2.17 


0.19 


1.03 


61 

U 1 


n 

u 


Hhcprup nhpnompna in a miantitat 1vp wav 
vuaci vc \ii iciiviiiwiia in a i lai i vc ivajr 


2.17 


0.19 


1.03 


72 


F 


Suoopst f oil ow-un 1 nvesti oat 1 ons 

93 1 V 1 1 VTT UU 1 IITWtfW 1 \£™ 


2.25 


0.07 


0.75 


uc 


n 

u 


PnnHncT* pvnpvifnpntc cafplw 

vUMUUCL CAyCl IIIICIILa aaicijr 


C iC3 


0.07 


0*87 


1Q 


F 


Annlv nrinrinlpc Inctpad of rntp fnrmtlap 
nysyiy \ii iiicipics iiiaivzau ui i uic luiiiiuiac 


C . J J 


\J * UU 


0 7R 


53 


D 


kppn a dav-to-dav lah dlarv 




-0*06 


1*23 


38 


B 


Construrt modpls which fit exn. pvidpncp 

VVIISwl U\t i. HMUC IS *flllt»ll lib \*nyj • CVI UCII^C 


2.33 


-0*06 


1*50 


98 


A 


Undprstand mpastirpjnpnt of diff. nhpnompna 

vnuci SLaiivi ii^asMi oiicih v ■ ui i i • i/irciijkAi^iia 


2.42 


-0.20 


1*08 


83 


F 


Hi sruss rpsults with othpr scientists 


2.42 


-0.20 


1.24 


52 


E 


Prnrpss p^npri mental data 


2.50 


-0.33 


1.09 


95 




Dpsinn rplpvant obsprvation tpchniaups 


2*50 


-0.33 


1.09 


101 


D 


Understand lah instructions 

vllVICI SiailVI IQU 1 11911 ULllUllJ 


2.58 


-0.45 


1.24 


29 


A 


Solve difficult scientific Droblems 


2.67 


-0.59 


1.07 


46 

•tu 


A 


Idpntifv variahlps & dptprminp emn relations 
xucii l i i jr vai i ou ICS oi vic lci iii i lie cii^j • i ciaii uii a 


2.67 


-0.59 


1.07 


28 


D 


Tarrv nut umivnt^ measurements 
v*ai i y uui ollui aic nicaaui ciiiciiLa 


? 75 


-0*72 


1.1 




n 


Pollert e^nerimental d^ta 


c »oj 


— v »o«j 


i.ii 


4 


D 


Usp nractical ^as onnosed to theoretical^ lah skills 




-0*99 


1 Oft 

1 »uv 


100 


E 


Confirm facts, prlnc* & theory from lect*/books 


3.00 


-1.11 


1.04 


12 


D 


Put basic lab* techniques to use 


3.08 


-1*24 


1.31 


88 


C 


Translate cone* def* Into set of meas* procedures 


3.08 


-1*24 


1.44 


24 


D 


Follow Instructions 


3.17 


-1*38 


1.27 


92 


D 


Know & apply altem* meas* techniques 


3.25 


-1.51 


1.22 


65 


E 


Confirm already known facts and laws 


3.42 


-1.77 


0.90 


18 


D 


Manipulate apparatl 


3.50 


-1.90 


1.24 


85 


D 


Handle modem equipment 


3.58 


-2.03 


1.08 


89 


D 


Set up lab equipment quickly & correctly 


3.67 


-2.17 


1.07 


32 


C 


Develop measurement techniques 


4.00 


-2.69 


1.13 


20 


D 


Calibrate instruments 


4.17 


-2.95 


0.72 
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Appendix 2 Order of priority of tad-terms from renesrch within tht OuN 



The first column gi*es the item ntmber for the specific end-term fron the Inventory (appendix 7). 
The roroan minerals 1n colum two correspond with the general era- tents. 



End-term 



s.d. 



03 II 
08 II 



14 
79 
90 
74 
49 
66 
15 



06 II 



37 
39 
57 
48 



27 II 
11 I 

81 II 
76 II 

50 II 

58 I 
67 I 
70 I 
78 II 

59 II 
02 II 

51 I 
30 II 
40 I 
69 II 
13 II 
99 I 
36 II 

82 I 
93 II 

60 II 
07 I 
73 II 
96 II 



Solve problems in a critical, academic way 

Approach observed phenomena from a sclent, point of view 

Hake decisions in proper course of action of prob-solving 

Have a critical attitude to exp. results 

Survey literature relevant to some problem 

Interpret data in literature 

Formulate a problem that can be researched 

Approach a problem with so open mind 

Form attitudes related to value & uses of exp. science 

Deeply understand the discipline studied 

Discover limitations of a theory/model 

Act independently & take initiative 

Apply one's insights, discoveries & conclusions 

Plan ahead 

Be Interested in the subject area 

Work in groups to solve sclent, problems 

Appreciate relationship between nature & science 

Design new exp. in their own fields 

Experience challenge of exp* method 

Be self-confident and independent 

Take active part in the process of science 

Work independently of others 

Experience spirit & essence of sclent, inquiry 

Build framework for facts, princ & theory from lect/books 

Use the lab as an Instrument for discovery 

Appreciate the usual & unusual 

Determine limits under which a theory applies 

Concretize theoretical notions 

Do experiments 

Illustrate facts, princ. & theory of lectures/books 
Use mental skills inherent to professionals 
Intuitively understand scientific phenomena 
Tackle a problem without help of others 
Experience kinship with the scientist 
Experience past and present scientists 1 joy 
Use motor skills inherent to professionals 
Experience joys & sorrows of experimenting 
Work in research & development labs 



17 
33 
50 
50 
50 
75 



1.83 
1.92 
2.00 
2.00 
2.00 



08 
25 
33 
33 
42 
42 
2.42 
2.50 
2.50 
2.50 
2.58 
2.58 
2.58 
2.67 
2.67 
2.67 
2.75 
2.83 
2.83 
2.83 
2.92 
2.92 
3.25 
3.25 
3.33 
3.33 
3.50 



16 



.59 
.59 
.59 
,15 
.01 
.86 
.72 
.72 
0.72 
0.58 
0.29 
0.15 
0.15 
-0.01 
-0.01 
-0.01 
-0.15 
-0.15 
-0.15 
-0.29 
-0.29 
-0.29 
-0.44 
-0.44 
-0.44 
-0.58 
-0.72 
-0.72 
-0.72 
-0.88 
-0.88 
-1.45 
-1.45 
-1.59 
-1.59 
-1.88 



39 
49 
52 
52 
90 
14 
72 
79 
43 
1.04 
1.21 
0.79 
0.75 
0.49 
1.15 
0.79 
1.00 
1.16 
0.90 
1.00 
1.09 
1.00 
1.00 
1.24 
0.89 
0.98 
1.37 
1.06 
0.94 
1.03 
1.03 
0.90 
1.08 
0.97 
1.06 
0.98 
0.98 
1.17 



^pp*ndix 3 Ordtr of priority of specific learning objtctivta from r •starch 
within UE 



The first column gives the Item number for the specific learning objective from the Inventory (appendix 7), 
The letters in the second column correspond with the general learning objectives. 







Objective 


x 


z 


s.d. 


62 


D 


Conduct experiments safely 


1.32 


1.54 


.0.48 


84 


A 


Understand the ouroose of an exDeriment 


1.32 


1.54 


0.48 


9 


E 


Interpret reliability and meaning of results 


1.36 


1.46 


0.49 


17 


F 


Present essentials of an exp. in written form 


1.36 


1.46 


0.49 


91 


A 


Understand what is to be measured in an exp* 


1.36 


1.46 


0.58 


25 


F 


Conrnunicate exD. findinas in written form 

W wJIVIIWI • 1 wtt WW W^Wr • III 1 V4 1 1 < J J III WW 9 | v bvl « 1 VI III 


1.41 


1.35 


0.50 


77 


c 


Properly plan an experiment 


1.50 


1.17 


0.51 


16 




Asspss relevance of exD. data with reoard to hvoothesis 


1.50 


1.17 


0.67 


64 




Rpronn hazards so as to takp safptv nrecautions 

F\w^\/UII. IIQfcQI 119 9\J OS kU balVC dB 1 C Wj LSI VVDUIIVH9 


1.50 


1 .17 

X . X / 


0.80 


87 


F 


Anal\/7P pxd. data to draw conclusions 


1.55 


1.06 


0.51 


47 


r 

c 


Pualtiatp pvn nutrpRv* with rpsnpct to a hvnothpsis 




n qr 

U . JO 


0.50 


9R 


n 

u 


PavY^v nut appitv^atp iMkacrtvomontc 




n qr 


n 

M . 39 


*JM 


p 

L. 


Pualtiatp rontri hut \ on oMrprt to rfprivpd Pi*i*ors 

LVQI UQ LC L>UI 1 LI 1 UU L 1 Mil U 1 1 CLL kU UCI 1 WCVi CI 1 Ml 9 


1 fi? 

X . Uc 


0 Q? 


0.74 


61 


D 


Ohsprx/p nhpnrmena in a Quantitative wav 


1,64 


0.87 


0.49 


4 


D 


Use practical (as opposed to theoretical) lab skills 


1.64 


0.87 


0.66 


80 


D 


Collect experimental data 


1.64 


0.87 


0.66 


102 


E 


Evaluate diff. expected & actual results 


1.68 


0.79 


0.57 


52 


E 


Process experimental data 


1.73 


0.69 


0.55 


71 


F 


Describe central aspects of an experiment 


1.73 


0.69 


0.55 


98 


A 


Understand measurement of diff* phenomena 


1.73 


0.69 


0.63 


34 


E 


Apply elementary notions of statistics 


1.73 


0.69 


0.77 


55 


F 


Sunmarize an exp. based on results 


1.76 


0.62 


0.70 


1 


E 


Hake order-of -magnitude calculations and estimates 


1.77 


0.60 


0.61 


101 


D 


Understand lab instructions 


1.82 


0.50 


0.91 


68 


D 


Handle waste safely 


1.82 


0.50 


1.05 


86 


c 


Understand scone & limits of exD. techniaues used 

VIIUCI 3 kallvl «bUvV W 1 1 fill VI WA|/« II 1 # %f MW« UJvU 


1.95 


0.23 


0.58 






llco oyn fiiftn to cnlvo cnorlf'fr nrohlpms 

Use cAp» uaia i,u 9Ui vc spct 1 I1C VIULJICttld 


1.95 




0.74 


31 




npsirm an pxn. to vprifv a theorv/hvnothesi s 


1.95 


0.23 


0.84 




u 


Pttt hasir lah tprhninups to iisp 


1.95 


0 ?3 

M . C J 


0.95 


Q7 


p 

L. 


Rplatp pyn otitrnmps to a narti ml ar thporv 

l\C IQLC CA^Ii UUiLU1IC3 LU a ysQl L 1 LU IQI LIICVIJF 


2.00 


0 1? 

M . It 


0.53 


5 


D 


nhspirvip nhpnnmpna in a nual \ tativp wax/ 
uuaci vc yJi icunipcua 111 a i^uai i La Live tfajr 


2.00 


0 1? 

M . It 


0 6? 


1 n 


P 

c 


Fc+imafo otitrmtP of pvn mpas within nivpn nrpri^ion 

LOl lllldLC UULL>UIIIC Ul CAU. IllCaa* HI llllll y I VCII UlCddlUII 


? tin 

C . MM 


M . It 


0 6? 

M . UC 


19 


E 


Ann] v nrincinles instead of rote formulae 


2.00 


0.12 


0.62 


35 


B 


Annlv eurrpnt knowledge in sol vino new nroblems 


2.05 


0.02 


0.95 


21 


A " 


Decomoose larae to smaller nroblems 


2.09 


-0.06 


0.75 


22 


c 


Desian subsequent exn. involvina Dhenomena 

WW d 1 VJI 1 JUUJVUUwl IW Wn W * 1JIVWI w I M M\J Wl 1 Wl IvyIPWI 111 


2.09 


-0.06 


0.75 


75 


B 


Apply known principles to new situations 


2.09 


-0.06 


0.75 


26 


E 


Use obtained data to make estimates in new situations 


2.14 


-0.17 


0.77 


72 


F 


Suggest follow-up investigations 


2.14 


-0.17 


0.77 


42 


F 


Conrnunicate exp. findings in oral form 


2.14 


-0.17 


0.94 


85 


D 


Handle modern equipment 


2.23 


-0.36 


0.87 


94 


B 


React adequately to unforeseen results 


2.23 


-0.36 


0.87 


23 


A 


Derive testable hypotheses from theories 


2.23 


-0.36 


1.02 


89 


D 


Set up lab equipment quickly & correctly 


2.27 


-0.44 


0.70 


18 


D 


Manipulate apparati 


2.27 


-0.44 


0.98 


53 


D 


keen a dav-to-dav lab diarv 


2.27 


-0.44 


1.20 


44 


D 


Be flexible in modifying exp* 


2.32 


-0.54 


1.09 




B 


fonstrnrt modpls haspd on pxd^ findinas 


2.41 


-0.73 


0.96 


38 


B 


Construct models which fit exp. evidence 


2.41 


-0.73 


1.05 


41 

tX 


« 

o 


Rpronni 7P & Hpf4np sripnt4f4r nroblpjns 

l\CLU\^l 1 1 *.C CM VIC 1 1 1 IC 9tl CI 11 1 1 1 L fit vU 1 CJHj 


? 41 

c . ~x 


u . / «J 


1.14 


OH 


A 


nov«4\iO Jt oualitato r»ol at 4 nnchl nc 
UCI IVC a CVdlUaLC 1 C 1 aL 1 Ullall 1 \I9 


C . HO 


n Aft 




DO 
OO 


r 

L 


Trancl af o rnnr Hpf in+n cot maac nv*nf*pHttv*oc 
li ailsldlc l>uiil>. uci . iiiLU scl Ul lllcaa. pi ulcuui ca 


C . 3U 




n ra 

u • ou 


92 


D 


Know & apply altern. meas. techniques 


2.50 


-0.92 


0.80 


46 


A 


Identify variables & determine emp. relations 


2.55 


-1.02 


0.96 


24 


D 


Follow instructions 


2.55 


-1.02 


1.06 


63 


A 


Solve problems in a multi-solution situation 


2.59 


-1.11 


1.05 


20 


D 


Calibrate instruments 


2.64 


-1.21 


1.05 


95 


C 


Design relevant observation techniques 


2.68 


-1.29 


0.99 


83 


F 


Discuss results with other scientists 


2.68 


-1.29 


1.13 


43 


E 


Incorporate unexpected exp. results in new model 


2.81 


-1.56 


1.08 


32 


C 


Develop measurement techniques 


3.00 


-1.96 


0.98 


29 


A 


Solve difficult scientific problems 


3.00 


-1.96 


1.38 


100 


E 


Confirm facts, princ. & theory from lect. /books 


3.24 


-2.46 


1.09 


65 


E 


Confirm already known facts and laws 


3.33 


-2.65 


0.91 
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Appendix 4 Ordtr of priority of specific ©nd-ttras froa r«s«arch within UE 

The first colunn gives the item nunber for the specific end-term from the Inventory (appendix 7). 
The roman numerals 1n column two correspond with the general end- terms. 



End-term x Z ».d. 

79 I Have a critical attitude to exp. results 1.45 2.09 0.51 

3 II Solve problems 1n a critical, academic way 1.64 1.67 0.58 

69 II Do experiments 1.73 1.47 1.08 

14 I Make decisions in proper course of action of prob-solving 1.86 1.18 0.71 

15 I Form attitudes related to value & uses of exp. science 1.86 1.18 0.89 
37 I Discover limitations of a theory/model 1.91 1.07 0.68 

6 II Deeply understand the discipline studied 1.91 1.07 0.75 
8 II Approach observed phenomena from a sclent, point of view 1.95 0.98 0.72 
2 II Use the lab as an instrument for discovery 2.oo 0.87 0.76 

48 I Plan ahead 2.05 0.76 0.72 

49 I Formulate a problem that can be researched 2.09 0.67 1.23 
74 I Interpret data in literature 2.14 0.56 0.71 

66 I Approach a problem with an open mind 2.14 0.56 1.08 
27 II Be interested in the subject area 2.18 0.47 1.10 
90 I Survey literature relevant to some problem 2.18 0.47 1.18 

39 I Act independently & take initiative 2.27 0.27 0.55 

57 I Apply one's insights, discoveries & conclusions 2.27 0.27 0.70 
76 II Design new exp. in their own fields 2.36 0.07 0.85 
78 II Experience spirit & essence of sclent, inquiry 2.36 0.07 0.95 
30 II Determine limits under which a theory applies 2.36 0.07 1.14 
99 I Use mental skills inherent to professionals 2.45 -0.13 1.06 
11 I Work in groups to solve sclent^ problems 2.45 -0.13 1.18 

81 II Appreciate relationship between nature & science 2.5o -0.24 1 n 6 
13 II Illustrate facts, princ. & theory of lectures/books 2.52 -0.29 1.03 
96 II Work in research & development labs 2.55 -0.35 1.30 

50 II Experience challenge of exp. method 2.59 -0.44 1.05 

40 I Concretize theoretical notions 2.62 -0.51 0.80 

67 I Take active part in the process of science 2.64 -0.55 1.09 
60 II Experience past and present scientists 1 joy 2.73 -0.75 1.24 

7 I Use motor skills inherent to professionals 2.81 -0.93 1.08 
70 I Work independently of others 2.86 -1.04 0.94 

51 I Appreciate the usual & unusual 2.91 -1.16 1.19 
73 II Experience joys & sorrows of experimenting 2.91 -1.16 1.44 

82 I Tackle a problem without help of others 3.05 -1.47 0.95 
36 II Intuitively understand scientific phenomena 3.05 -1.47 1.18 

58 I Be self-confident and independent 3.14 -1.67 1.17 
93 II Experience kinship with the scientist 3.18 -1.76 1.10 

59 II Build framework for facts, princ & theory from lect/books 3.19 -1.78 1.08 
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Appendix 5 List of participating institutes of higher distance education 

Athabasca University, Canada 

Teie-Universite (TELUQ), University de Quebec, Canada 

Centre for Off-campus Studies, Universiti Sains Malaysia (Science University 
of Malaysia). Malaysia 

University of Queensland, Australia 

Empire State College, New York, USA 

British Open University, Milton Keynes, GB 

Sukhothai Thammathirat Open University (STOU), Thailand 

Open University Sri Lanka, Sri Lanka 

Everyman's University, Tel Aviv, Israel 

University of Wisconsin Superior, Wisconsin, USA 



ERLC 



42 



Appendix 6 A paired comparison of general objectives, LDSI-2 



LDSI-2 



Name: 



A practical is defined as those activities relating to experimentation 
beginning with the conception of a question or the observation of a phenomenon 
through the reporting of the results in written or oral form. Examples of 
experimentation are: demonstrations, , wet' labs, pen and paper experiments, 
simulations, etc* 

This booklet contains 28 pairs of general learning objectives for practicals. 
Please choose the objective by each pair which you consider the most important 
of the two. You can do this by placing a cross (x) in the circle preceding the 
relevant objective • Please do not look back to or chang e previous choicest 
Thank You. 



LDSI-2 



After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 use knowledge and skills in unfamiliar situations 
0 interpret experimental data 

After completing an undergraduate study in the Natural Sciences 9 a student 
should be able to: 

0 remember the central idea of an experiment over a significantly 

long period of time 
0 design (simple) experiments to test hypotheses 

After completing an undergraduate study in the Natural Sciences 9 a student 
should be able to: 

0 formulate hypotheses 
0 solve problems 

After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 design (simple) experiments to test hypotheses 
0 use knowledge and skills in unfamiliar situations 

After completing an undergraduate study in the Natural Sciences* a student 
should be able to: 

0 interpret experimental data 

0 design (simple) experiments to test hypotheses 

After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 clearly describe an expsriment 

0 remember the central idea of an experiment over a significantly 
long period of time 



After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 remember the central idea of an experiment over a significantly 

long period of time 
0 use laboratory skills in performing (simple) experiments 
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LDSI-2 



After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 clearly describe an experiment 
0 solve problems 

After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 remember the central idea of an experiment over a significantly 

long period of time 
0 formulate hypotheses 



After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 interpret experimental data 

0 use laboratory skills in performing (simple) experiments 

After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 solve problems 

0 use laboratory skills in performing (simple) sxperiments 



After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 formulate hypotheses 

0 interpret experimental data 



After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 formulate hypotheses 

0 design (simple) experiments to test hypotheses 



After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 clearly describe an experiment 

0 use knowledge and skills in unfamiliar situations 
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After completing an undergraduate study in the Natural Sciences* a student 
should be able to: 

0 clearly describe an experiment 
0 interpret experimental data 

After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 design (simple) experiments to test hypotheses 
0 clearly describe an experiment 

After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 formulate hypotheses 

0 use laboratory skills in performing (simple) experiments 

After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 interpret experimental data 

0 remember the central idea of an experiment over a significantly 
long period of time 

After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 remember the central idea of an experiment over a significantly 

long period of time 
0 solve problems 

After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 solve problems 

0 interpret experimental data 

After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 solve problems 

0 design (simple) experiments to test hypotheses 
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LDSI-2 



After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 clearly describe an experiment 

0 use laboratory skills in performing (simple) experiments 

After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 use knowledge and skills in unfamiliar situations 
0 remember the central idea of an experiment over a significantly 
long period of time 

After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 use knowledge and skills in unfamiliar situations 
0 formulate hypotheses 

After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 use laboratory skills in performing (simple) experiments 
0 use knowledge and skills in unfamiliar situations 

After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 solve problems 

0 use knowledge and skills in unfamiliar situations 

After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 clearly describe an experiment 
0 formulate hypotheses 



After completing an undergraduate study in the Natural Sciences, a student 
should be able to: 

0 design (simple) experiments to test hypotheses 

0 use laboratory skills in performing (simple) experiments 
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Appendix 8 List of general objectives and end-terms with the corresponding 
specific learning objectives and end-terms. 

This classification is based upon the classification of specific learning objectives and end-terms by Meester 
and Klrschner (1988). The number of general learning objectives has been reduced from eight to six. 
Several specific learning objectives have been reformulated; four specific learning objectives have become 
end- terms, and four end-terms have become specific learning objectives. 

N.B.: the numbers preceding the specific learning objectives and end-terms correspond with the item number 
from the inventory (appendix 7). 



Specific learning objectives 



A. To solve problems 

21 decompose large problems into a number of smaller problems 

23 derive testable hypotheses from theories 

63 solve problems in which there is more than one usable solution strategy 
54 derive and evaluate relationships between observed scientific phenomena 
33 use experimental data to solve specific problems 

29 solve difficult scientific problems 

64 understand the purpose of an experiment 

91 understand what is to be measured during an experiment 

98 understand how different phenomena are measured during an experiment 

46 identify the variables that adequately describe some system's state and 
empirically determine the way they are related 



B. To use knowledge and skills in unfamiliar situations 

35 apply what is already known to solve new problems 

75 apply known principles to new situations 

41 recognize and define scientific problems 

45 construct models based on experimental findings 

38 construct models which fit experimental evidence 

94 react adequately when confronted with unforeseen results 



C. To design (simple) experiments to test hypotheses 

31 design an experiment to test a theory or hypothesis 
77 properly plan an experiment 

95 design observation techniques relevant to the task at hand 

32 develop measurement techniques 

22 design subsequent experiments involving the phenomena being studied 
64 recognize hazards so as to take appropriate safety precautions 
86 understand the scope and limiting conditions of the experimental techniques, 
used 

88 translate a conceptual definition of a quantity into a set of measurement 
procedures 
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D. To use laboratory skills in performing (simple) experiments 



24 follow instructions 

101 understand laboratory instructions 

4 use practical (as opposed to theoretical) laboratory skills 
89 set up laboratory equipment quickly and correctly 

18 manipulate apparati 

62 conduct experiments safely 

92 apply alternative measuring techniques for improving reliability and 

precision of data gained 
12 put basic laboratory techniques (such as titration, microscopy or 

physical measurement) to use 
85 handle modern equipment 
20 calibrate instruments 
28 carry out accurate measurements 

5 observe phenomena in a qualitative way 
61 observe phenomena in a quantitative vay 

44 be flexible with respect to modifying experiments in light of results 

obtained in prior experimentation 
68 handle waste safely from an environmental point of view 
80 collect experimental data 

53 keep a day-to-day laboratory diary in such a way that a third person can 
repeat the experiments 



E. To interpret experimental data 
52 process experimental data 

87 analyse experimental data in order to draw conclusions from them 

19 apply principles rather than rote use of computational formulae in the 

theoretical analysis of the lab experiment 
34 apply elementary notions of statistics (e.g. random errors, systematic 

errors, mean values, uncertainty and confidence limits) in evaluating 

experimental data 

56 evaluate how errors in direct measurements may contribute to errors in a 
derived measure 

16 assess the relevance of experimental data with regard to a hypothesis being 
studied 

10 estimate the outcome of experimental measurements within a given precision 

prior to actual experimentation 
47 evaluate the outcome of an experiment with regard to the hypothesis being 

tested 

1 make order-of -magnitude calculations and estimates 
43 incorporate unexpected experimental results in a new model 
102 evaluate why expected (theoretical) results differ from actual experimental 
findings 

26 use data already obtained to make estimates regarding not yet tested 
situations 

9 interpret the reliability and meaning of results gained through 
experimentation (either their own or those of others) 
97 relate the outcomes of an experiment to a particular theory 
100 confirm facts, principles and theories discussed in lectures or books 
65 confirm already known facts and laws 
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F. To clearly describe the experiment 

17 present the essentials of an experiment in written form 

55 summarize the important aspects of an experiment based on collected data 

71 describe the central aspects of an experiment (i.e. its goals, underlying 
theory and basic methods) 

25 communicate experimental findings in written form 
42 communicate experimental findings in oral form 

72 suggest follow-up investigations once the results of a scientific 
investigation are known 

83 discuss results of scientific investigations with other scientists 



End-terms 



I. To obtain good scientific attitudes 



49 formulate a problem so that it can be researched 

90 survey the literature relevant to some problem at hand 

14 make decisions as to the proper course of action in solving problems 
79 have a critical attitude towards experimentally gained results 

58 be self-confident and independent 

39 act independently and take initiative 

82 tackle a problem without the help of others 

48 plan ahead 

40 concretize (illustrate) theoretical notions 

15 form attitudes relating to the value and uses of experimental science 
(physics, biology and chemistry) 

57 apply one's own insights * discoveries and conclusions in explaining 

observed phenomena 

37 discover the limitations of a theory or model 

66 approach a problem with an open mind 

11 work in groups to solve scientific problems 

70 work independently of others 

67 actively take part in the process of science 

99 use mental skills inherent to professionals in the natural sciences 

74 interpret data in the literature 

7 use motor skills inherent to professionals in the natural sciences 

51 appreciate the usual as veil as the unusual within the natural sciences 
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II. To understand the scientific method 

81 appreciate the relationship between nature and science 

36 intuitively understand scientific phenomena 

30 determine the limits tinder which a theory applies 

69 do experiments 

13 illustrate facts, principles and theories discussed in lectures or books 

59 build a framework for facts, principles and theories encountered in 
lectures and books 

2 use the laboratory as an instrument for discovery 
96 work in research and developments laboratories 

8 approach observed phenomena from a scientific point of view 

50 experience the intellectual challenge of using the experimental method 

73 experience the joys and sorrows of experimenting 

93 experience a kinship with the scientist 

60 experience the joy experienced by scientists past and present 

6 deeply understand the discipline studied (as opposed to being able to 
simply work by the book) 

78 experience the spirit of scientific inquiry and the essence of scientific 
thinking 

27 be interested in the subject area being studied, 

76 design new experiments in their own field of research 

3 solve problems in a critical, academic way 



Appendix 9 x, Z and s.d. of specific learning objectives for DUE institutes 



The first coltmn givesthe Item nutter for the specific learning objective from the Inventory (appendix 7). 
The letters 1n the second column correspond with the general learning objectives. 



Objective 



s.d. 



87 E Analyze exp. data to draw conclusions 1*33 1.82 

9 E Interpret reliability and meaning of results 1.4 1.67 

91 A Understand what 1s to be measured 1n an exp. 1.4 1.67 

84 A Understand the purpose of an experiment 1.53 1.39 

62 D Conduct experiments safely 1.57 1.30 

19 E Apply principles instead of rote formulae 1.6 1.24 

61 D Observe phenomena 1n a quantitative way 1.6 1.24 

1 E Make order-of -magnitude calculations and estimates 1.6 1.24 

4 D Use practical (as opposed to theoretical) lab skills 1.67 1.09 

25 F Communicate exp. findings in written form 1.67 1.09 

47 E Evaluate exp. outcome with respect to a hypothesis 1.67 1.09 

64 C Recog>*> hazards so as to take safety precautions 1.73 0.95 

80 D Collect experimental data 1.73 0.95 

86 C Understand scope & limits of exp. techniques used 1.8 0.80 

98 A Understand measurement of diff. phenomena 1.8 0.80 

17 F Present essentials of an exp. in written form 1.8 0.80 
101 D Understand lab instructions 1.8 0.80 

16 E Assess relevance of exp. data with regard to hypothesis 1.87 0.65 

52 E Process experimental data 1.87 0,65 

77 C Properly plan an experiment 1.87 0.65 

Apply current knowledge in solving new problems 1.93 0.52 

Handle waste safely 1*93 0.52 

Evaluate contribution direct to derived errors 1.93 0.52 

Apply known principles to new situations 1*93 0.52 

Describe central aspects of an experiment 1.93 0.52 

Sirnmarize an exp. based on results 1.93 0.52 

Evaluate diff. expected & actual results 1.93 0.52 

Carry out accurate measurements 1.93 0.52 

Put basic lab. techniques to use 2.00 0.37 

Observe phenomena in a qualitative way 2.00 0.37 

Derive & evaluate relationships 2.00 0.37 

Apply elementary notions of statistic 2.00 0.37 

Relate exp. outcomes to a particular theory 2.13 0.09 

41 B Recognize & define scientific problems 2.13 0.09 

95 C Design relevant observation techniques 2.20 -0.07 

21 A Decompose large to smaller problems 2.27 -0.22 

23 A Derive testable hypotheses from theories 2.27 -0.22 

33 A Use exp. data to solve specific problems 2.33 -0.35 

18 D Manipulate apparati 2.33 -0.35 
94 B React adequately to unforeseen results 2.33 -0.35 

Use obtained data to make estimates in new situations 2.33 -0.35 

Discuss results with other scientists 2.36 -0.41 

Design an exp. to verify a theory/hypothesis 2.40 -0.50 

Identify variables & determine emp. relations 2.40 -0.50 

Be flexible in modifying exp. 2.40 -0.50 

Follow instructions 2.40 -0.50 

Communicate exp. findings in oral form 2.47 -0.65 

Know & apply altem. meas. techniques 2.53 -0.78 

Solve problems in a multi -solution situation 2.53 -0.78 

Design subsequent exp. involving phenomena 2.50 -0.78 

Translate cone, def . into set of meas. procedures 2.60 -0.94 

Estimate outcome of exp. meas. within given precision 2.67 -1.09 

Suggest follow-up investigations 2.73 -1.22 

Keep a day-to-day lab diary 2.73 -1.22 

Confirm facts, princ. & theory from lect. /books 2.73 -1.22 

Handle modern equipment 2.80 -1.37 

Set up lab equipment quickly & correctly 2.80 -1.37 

38 8 Construct models which fit exp. evidence 2.87 -1.52 

65 E Confirm already known facts and laws 2.87 -1.52 

32 C Develop measurement techniques 2.87 -1.52 

45 8 Construct models based on exp. findings 2.93 -1.65 

43 E Incorporate unexpected exp. results in new model 3.00 -1.81 

29 A Solve difficult scientific problem 3.07 -1.96 

20 D Calibrate Instruments 3.07 -1.96 



35 8 

68 D 

56 E 

75 B 

71 F 

55 F 

102 E 

28 D 

12 D 

5 D 

54 A 

34 E 

97 E 



26 
83 
31 
46 
44 
24 
42 
92 
63 
22 
88 
10 
72 
53 
100 
85 
89 



0.49 
0.51 
0.83 
0.74 
0.65 
0.51 
0.51 
0.63 
0.62 
0.82 
0.90 
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68 
68 
86 
08 
92 
0.92 
0.92 
0.80 
0.80 
0.92 
1.03 
1.10 
1.16 
1.16 
1.22 
0.38 
0.76 
0.76 
1.00 
0.74 
1.06 
1.08 
0.70 
0.96 
0.62 
0.72 
0.72 
0.98 
1.08 
1.06 
1.12 
1.40 
1.45 
0.92 
0.64 
0.83 
0.99 
0.83 
1.18 
1.03 
1.22 
1.22 
1.08 
1.08 
1.13 
1.13 
1.19 
0.96 
1.13 
1.10 
1.28 
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Appendix 10 x, Z and s.d. of specific end-terms for DUE institutes 

The first column gives the Item nuitoer for the specific end-term from the Inventory (appendix 7). 
The reman minerals 1n col urn two correspond with the general end- terms. 



End-term 



s.d. 



8 


II 


79 


I 


3 


II 


66 


I 


14 


I 


6 


II 


69 


II 


15 


I 


90 


I 


48 


I 


78 


II 


2 


II 


99 


I 


74 


I 


13 


II 


59 


II 


40 


I 


81 


II 


27 


II 


39 


I 


49 


I 


50 


II 


£7 


I 


57 


I 


51 


I 


82 


I 


7 


I 


76 


II 


60 


II 


37 


I 


73 


II 


30 


II 


58 


I 


11 


I 


36 


II 


70 


I 


96 


II 


93 


II 



Approach observed phenomena from a sclent, point of view 1.53 

Have a critical attitude to exp. results 1.53 

Solve problems 1n a critical, academic way 1.71 

Approach a problem with an open mind 1.73 
Hake decisions in proper course of action of prob-solving 1.80 

Deeply understand the discipline studied 1*87 

Do experiments 1*87 

Form attitudes related to value & uses of exp. science 2.00 

Survey literature relevant to soue problem 2.13 

Plan ahead 2 » 13 

Experience spirit & essence of sclent. Inquiry 2.13 

Use the lab as an Instrument for discovery 2.13 

Use mental skills inherent to professionals 2.14 

Interpret data In literature 2.20 

Illustrate facts, prlnc. & theory of lectures/books 2.20 
Build framework for facts, prlnc & theory from lect/books 2.20 

Concretize theoretical notions 2.29 

Appreciate relationship between nature & science 2.33 

Be Interested 1n the subject area 2.33 

Act independently & take initiative 2.47 

Formulate a problem that can be researched 2.47 

Experience challenge of exp. method 2.47 

Take active part in the process of science 2.57 

Apply one's insights, discoveries & conclusions 2.6 

Appreciate the usual & unusual 2.67 

Tackle a problem without help of others 2.67 

Use motor skills inherent to professionals 2.67 

Design new exp. in their own fields 2.67 

Experience past and present scientists 1 joy 2.71 

Discover limitations of a theory/model 2.73 

Experience joys & sorrows of experimenting 2.73 

Determine limits under which a theory applies 2.80 

Be self-confident and independent 2.80 

Work Sn groups to solve sclent, problems 2.93 

Intuitively understand scientific phenomena 2.93 

Work Independently of others 3.13 

Work in research & development labs 3.33 

Experience kinship with the scientist 3.53 



.86 
.86 
.48 
.43 
.28 
.13 
.13 
0.85 
0.57 
0.57 
0.57 
0.57 
0.55 
0.42 
0.42 
0.42 
0.23 
0.15 
0.15 
-0.15 
-0.15 
-0.15 
-0.37 
-0.43 
-0.58 
-0.58 
-0.58 
-0.58 
-0.67 
-0.71 
-0.71 
-0.86 
-0.86 
-1.14 
-1.14 
-1.57 
-2.00 
-2.43 



0.52 

0.52 

0.47 

0.88 

0.68 

0.74 

0.92 

0.65 

0.92 

0.99 

1.06 

1.13 

1.29 

0.86 

0.94 

1.01 

0.73 

0.82 

1.11 

1.13 

1.13 

1.19 

1.09 

0.91 

0.90 

1.05 

1.18 

1.29 

1.44 

1.10 

1.16 

1.26 

1.26 

1.16 

1.21 

1.06 

1.18 

0.99 
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Appendix 11 



Order of magnitude of differences in normalised learning objec- 
tive scores between DUE and UE 



The first colunn gives the item ntiriber for the specific learning objective from the Inventory (appendix 7), 
The letters 1n the second column correspond with the general learning objectives. 

Objective DUE UE hi 

54 A Derive & evaluate relationships 0.37 -0.88 1.25 

100 E Confirm facts, princ. & theory from lect. /books -1.22 -2.46 1.24 

95 C Design relevant observation techniques -0.07 -1.29 1.22 
10 E Estimate outcome of exp. meas. within given precision -1.09 0.12 1.21 

65 E Confirm already known facts and laws -1.52 -2.65 1.13 

19 E Apply principles Instead of rote formulae 1.24 0.12 1.12 
72 F Suggest follow-up Investigations -1.22 -0.17 1.05 
B5 D Handle modern equipment -1.37 -0.36 1.01 
B9 D Set up lab equipment quickly & correctly -0.44 -1.37 0.93 

45 B Construct models based on exp. findings -0.73 -1.65 0.92 
B3 F Discuss results with other scientists -1.29 -0.41 0.88 

41 8 Recognize & define scientific problems 0.09 -0.73 0.82 
38 8 Construct models which fit exp. evidence -1.52 -0.73 0.79 
53 D Keep a day-to-day lab diary -1.22 -0.44 0.78 
87 E Analyze exp. data to draw conclusions 1.82 1.06 0.76 

20 D Calibrate instruments -1.96 -1.21 0.75 

31 C Design an exp. to verify a theory/hypothesis -0.50 0.23 0.73 
22 C Design subsequent exp. Involving phenomena -0.78 -0.06 0.72 
17 F Present essentials of an exp. 1n written form 0.80 1.46 0.66 

1 E Make order-of -magnitude calculations and estimates 1.24 0.60 0.64 

33 A Use exp. data to solve specific problems -0.35 0.23 0.58 
75 B Apply known principles to new situations 0.52 -0.06 0.58 
86 C Understand scope & limits of exp. techniques used 0.80 0.23 0.57 

24 D Follow instructions -0.50 -1.02 0.52 

46 A Identify variables & determine emp. relations -0.50 -1.02 0.52 
16 E Assess relevance of exp. data with regard to hypothesis 0.65 1.17 0.52 
77 C Properly plan an experiment 0.65 1.17 0.52 
35 8 Apply current knowledge in solving new problems 0.52 0.02 0.50 

42 F Comnunicate exp. findings in oral form -0.65 -0.17 0.48 
28 D Carry out accurate measurements 0.52 0.98 0.46 

32 C Develop measurement techniques -1.52 -1.96 0.44 
56 E Evaluate contribution direct to derived errors 0.52 0.92 0.40 

61 D Observe phenomena in a quantitative way 1.24 0.87 0.37 

63 A Solve problems in a multi-solution situation -0.78 -1.11 0.33 

34 E Apply elementary notions of statistics 0.37 0.69 0.32 

101 D Understand lab instructions 0.80 0.50 0.30 

102 E Evaluate diff. expected & actual results 0.52 0.79 0.27 

25 F Communicate exp. findings in written form 1.09 1.35 0.26 
5 D Observe phenomena in a qualitative way 0.37 0.12 0.25 

43 E Incorporate unexpected exp. results in new model -1.B1 -1.56 0.25 

62 D Conduct experiments safely 1.30 1.54 0.24 
4 D Use practical (as opposed to theoretical) lab skills 1.09 0.87 0.22 

64 C Recogn. hazards so as to take safety precautions 0.95 1.17 0.22 
9 E Interpret reliability and meaning of results 1.67 1.46 0.21 

91 A Understand what is to be measured in an exp. 1.46 1.67 0.21 

26 E Use obtained data to make estimates in new situations -0.35 -0.17 0.18 
71 F Describe central aspects of an experiment 0.52 0.69 0.17 
21 A Decompose large to smaller problems -0.22 -0.06 0.16 
84 A Understand the purpose of an experiment 1.39 1.54 0.15 

92 D Know & apply altern. meas. techniques -0.78 -0.92 0.14 
12 D Put basic lab. techniques to use 0.37 0.23 0.14 
23 A Derive testable hypotheses from theories -0.22 -0.36 0.14 
47 E Evaluate exp. outcome with respect to a hypothesis 1.09 0.98 0.11 
98 A Understand measurement of diff. phenomena 0.80 0.69 0.11 
55 F Sunmarize an exp. based on results 0.52 0.62 0.10 
18 D Manipulate apparati -0.35 -0.44 0.09 
80 D Collect experimental data 0.95 0.87 0.08 

44 D Be flexible in modifying exp. -0.50 -0.54 0.04 
52 E Process experimental data 0.65 0.69 0.04 
97 E Relate exp. outcomes to a particular theory 0.09 0.12 0.03 
68 D Handle waste safely 0.52 0.50 0.02 
88 C Translate cone. def . into set of meas. procedures -0.94 -0.92 0.02 
94 B React adequately to unforeseen results -0.35 -0.36 0.01 
29 A Solve difficult scientific problems -1.96 -1.96 0.00 

n.b.: AZ is an absolute scot^ 
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Appendix 12 Ordtr of magnitude of difftrtncts in normaliztd Iteming obj«c- 
tivt scorts bttwttn OuN and DUE 



The first colimn gives the Item number for the specific learning objective from the inventory (appendix 7), 
The letters in the second column correspond with the general learning objectives. 



Objective 

43 E Incorporate unexpected exp. results in new model 

4 D Use practical (as opposed to theoretical) lab skills 
45 B Construct models based on exp. findings 
80 D Collect experimental data 
12 D Put basic lab. techniques to use 
10 IE Estimate outcome of exp. meas. within given precision 

18 D Manipulate apparati 

38 B Construct models which fit exp. evidence 
29 A Solve difficult scientific problems 

22 C Design subsequent exp. involving phenomena 

19 E Apply principles instead of rote formulae 
72 F Suggest follow-up investigations 

101 D Understand lab Instructions 
28 D Carry out accurate measurements 
63 A Solve problems in a multi -solution situation 
62 D Conduct experiments safely 
21 A Decompose large to smaller problems 

23 A Derive testable hypotheses from theories 

32 C Develop measurement techniques 
53 D Keep a day-to-day lab diary 

87 E Analyze exp. data to draw conclusions 

44 D Be flexible In modifying exp. 

94 B React adequately to unforeseen 

61 D Observe phenomena in a quantitative way 
98 A Understand measurement of diff. phenomena 

20 D Calibrate instruments 

42 F Cownunicate exp. findings 1n oral form 
52 E Process experimental data 
31 C Design an exp. to verify a hypothesis/theory 
97 E Relate exp. outcomes to a particular theory 

24 D Follow instructions 

34 E Apply elementary notions of statistics 

26 E Use obtained data to make estimates 1n new situations 

33 A Use exp. data to solve specific problems 
89 D Set up lab equipment quickly & correctly 
92 D Know & apply altern. meas. techniques 

91 A Understand what is to be measured in an exp. 

85 D Handle modern equipment 

16 E Assess relevance of exp. data with regard to hypothesis 
84 A Understand the purpose of an experiment 

25 F Comtwnicate exp. findings in written form 

86 C Understand scope & limits of exp. techniques used 
71 F Describe central aspects of an experiment 

102 E Evaluate diff. expected & actual results 
1 E Make order-of -magnitude calculations and estimates 
41 B Recognize & define scientific problems 
47 E Evaluate exp. outcome with respect to a hypothesis 
5 D Observe phenomena in a qualitative way 
68 D Handle waste safely 

88 C Translate cone. def. into set of meas. procedures 

95 C Design relevant observation techniques 
65 E Confirm already known facts and laws 

64 C Recogn. hazards so as to take safety precautions 
83 F Discuss results with other scientists 

54 A Derive & evaluate relationships 

75 B Apply known principles to new situations 

55 F Summarize an exp. based on results 

56 E Evaluate contribution direct to derived errors 
77 C Properly plan an experiment 

100 E Confirm facts, princ. & theory from lect. /books 
46 A Identify variables & determine emp. relations 

17 F Present essentials of an exp. in written form 
35 B Apply current knowledge in solving new problems 

9 E Interpret reliability and meaning of results 



OuN DUE LI 



0.58 * 
-0.99 
0.19 • 
-0.85 
-1.24 
0.46 
-1.90 
-0.06 
-0.59 
0.58 
-0.06 
0.07 
-0.45 
-0.72 
0.46 
0.07 
0.98 
0.98 
-2.69 
-0.06 
0.73 
0.58 
0.73 
0.19 
-0.20 
-2.95 
0.33 
0.33 
0.46 
0.98 
-1.38 
1.24 
0.46 
0.46 
-2.17 
-1.51 
0.98 
-2.02 
1.24 
0.85 
0.58 
0.33 
0,98 
0.98 
0.85 
0.46 
0.73 
0.73 
0.19 
-1.24 
-0.33 
-1.77 
0.73 
-0.20 
0.58 
0.73 
0.33 
0.33 
0.46 
-1.11 
-0.59 
0.73 
0.46 
1.64 



1.81 
1.09 
1.65 
0.95 
0.37 
•1.09 
0.35 
•1.52 
•1.96 
•0.78 
1.24 
•1.22 
0.80 
0.52 
-0.78 
1.30 
•0.22 
-0.22 
-1.52 



22 
82 
05 
35 
24 
80 
-1.96 
-0.65 
0.65 
-0.50 
0.09 
-0.50 
0.37 
-0.35 
-0.35 
-1.37 
-0.78 
1.67 
-1.37 
0.65 
1.39 
1.09 
0.80 
0.52 
0.52 
1.24 
0.09 
1.09 
0.37 
0.52 
-0.94 
-0.07 
-1.52 
0.95 
-0.41 
0.37 
0.52 
0.52 
0.52 
0.65 
-1.22 
-0.50 
0.80 
0.52 
1.67 



2.39 

2.08 

1.84 

1.80 

1.61 

1.55 

1.55 

1.46 

1.37 

1.36 

1.30 

1.29 

1.25 

1.24 

1.24 

1.23 

1.20 

1.20 

1.17 

1.16 

1.09 

1.08 

1.08 

1.05 

1.00 

0.99 

0.98 

0.98 

0.96 

0.89 

0.88 

0.87 

0.81 

0.81 

0.80 

0.73 

0.69 

0.65 

0.59 

0.54 

0.51 

0.47 

0.46 

0.46 

0.39 

0.37 

0.36 

0.36 

0.33 

0.30 

0.26 

0.25 

0.22 

0.21 

0.21 

0.21 

0.19 

0.19 

0.19 

0.11 

0.09 

0.07 

0.06 

0.03 



n.b.: aZ is an absolute score 



Appendix 13 Order of magnitude of differences in normalised learning objec 
tive arcoree between OuN and UE 



The first column gives the Item number for the specific learning objective from the inventory (appendix 7). 
The letters in the second column correspond with the general learning objectives. 



Objective OuN UE AZ 

43 E Incorporate unexpected exp. results in new model 0.58 -1.56 2.14 
4 D Use practical (as opposed to theoretical) lab skills -0.99 0.87 1.86 

20 D Calibrate instruments -2.95 -1.21 1.74 
89 D Set up lab equipment quickly & correctly -2.17 -0.44 1.73 
80 D Collect experimental data -0.85 0.87 1.72 

28 D Carry out accurate measurements -0.72 0.98 1.70 
85 D Handle modern equipment -2.02 -0.36 1.66 
63 A Solve problems in a multl -solution situation 0.46 -1.11 1.57 
12 D Put basic lab. techniques to use -1.24 0.23 1.47 
62 D Conduct experiments safely 0.07 1.54 1.47 
18 D Manipulate apparati -1.90 -0.44 1.46 
54 A Derive & evaluate relationships 0.58 -0.88 1.46 

29 A Solve difficult scientific problems -0.59 -1.96 1.37 

100 E Confirm facts, princ. & theory from lect. /books -1.11 -2.46 1.35 
23 A Derive testable hypotheses from theories 0.98 -0.36 1.34 
41 B Recognize & define scientific problems 0.46 -0.73 1.19 

44 D Be flexible in modifying exp. 0.58 -0.54 1.12 

83 F Discuss results with other scientists -O.tO -1.29 1.09 

94 B React adequately to unforeseen results 0.73 -0.36 1.09 

21 A Decompose large to smaller problems 0.98 -0.06 1.04 
52 E Process experimental data -0.33 0.69 1.02 

95 C Design relevant observation techniques -0.33 -1.29 0.96 

101 D Understand lab instructions -0.45 0.50 0.95 

45 B Construct models based on exp. findings 0.19 -0.73 0.9Z 
98 A Understand measurement of diff. phenomena -0.20 0.69 0.89 
65 E Confirm already known facts and laws -1.77 -2.65 0.88 
97 E Relate exp. outcomes to a particular theory 0.98 0.12 0.86 
75 B Apply known principles co new situations 0.73 -0.06 0.79 

25 t Communicate exp. findings in written form 0.58 1.35 0.77 
17 F Present essentials of an exp. in written form 0.73 1.46 0.73 
32 C Develop measurement techniques -2.69 -1.96 0.73 
77 C Properly plan an experiment 0.46 1.17 0.71 

84 A Understand the purpose of an experiment 0.85 1.54 0.69 
61 D Observe phenomena in a quantitative way 0.19 0.87 0.68 
38 B Construct models which fit exp. evidence -0.06 -0.73 0.67 

22 C Design subsequent exp. involving phenomena 0.58-0.06 0.64 

26 E Use obtained data to make estimates 1n new situations 0.46 -0.17 0.63 
5 D Observe phenomena in a qualitative way 0.73 0.12 0.61 

56 E Evaluate contribution direct to derived errors 0.33 0.92 0.59 

92 D Know & apply altern. meas. techniques -1.51 -0.92 0.59 

34 E Apply elementary notions of statistics 1.24 0.69 0.55 
42 F Communicate exp. findings 1n oral form 0.33 -0.17 0.50 
91 A Understand what is to be measured 1n an exp. 0.98 1.46 0.48 

35 B Apply current knowledge in solving new problems 0.46 0.02 0.44 
64 C Recogn. hazards so as to take safety precautions 0.73 1.17 0.44 

46 A Identify variables & determine emp. relations -0.59 -1.02 0.43 
53 D Keep a d^y-to-day lab diary -0.06 -0.44 0.38 
24 D Follow Instructions -1-38 -1.02 0.36 
10 E Estimate outcome of exp. meas. witiln given precision 0.46 0.12 0.34 

87 E Analyze exp. data to draw conclusions 0.73 1.06 0.33 

88 C Translate cone, def . into set of meas. procedures -1.24 -0.92 0.32 
68 D Handle waste safely 0.19 0.50 0.31 
55 F Sunrnarize an exp. based on results 0.33 0.62 0.29 

71 F Describe central aspects of an experiment 0.98 0.69 0.29 
1 E Make order-of-magnitude calculations and estimates 0.85 0.60 0.25 

47 E Evaluate exp. outcome with respect to a hypothesis 0.73 0.98 0.25 

72 F Suggest follow-up investigations 0.07 -0.17 0.24 
31 C Design an exp. to verify a theory/hypothesis 0.46 0.23 0.23 
33 A Use exp. data to solve specific problems 0.46 0.23 0.23 

102 E Evaluate diff. expected & actual results 0.98 0.79 0.19 

9 E Interpret reliability and meaning of results 1.64 1.46 0.18 

19 E Apply principles Instead of rote formulae -0.06 0.12 0.18 

86 C Understand scope & limits of exp. techniques used 0.33 0.23 0.10 

16 E Assess relevance of exp. data with regard to hypothesis 1.24 1.17 0.07 



n.b.: AZ is an absolute score 



Appendix 14 Ordar of magnituda of diffarancas in normalised and-tarm acoraa 
batwaan DUE and UE 



The first column gives the Item number for the specific end- term from the Inventory (appendix 7). 
The roman numerals 1n column two correspond with the general end-terms. 
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II 



Build framework for facts, prlnc & theory from lect/books 0,42 -1.78 2,20 

Discover limitations of a theory/model -0,71 1.07 1,78' 

Work 1n research & development labs -2,00 -0,35 1,65 

Work 1n groups to solve sclent, problems -1.14 -0,13 1.01 

Determine limits under which a theory applies -0.86 0.07 0.93 

Tackle a problem without help of others -0.58 -1.47 0.89 

Approach observed phenomena from a sclent, point of view 1.86 0.98 0.88 

Approach a problem with an open mind 1.43 0.56 0.87 

Formulate a problem that can be researched -0.15 0.67 0.82 

Be self-confident and Independent -0.86 -1.67 0.81 

Concretize theoretical notions 0.23 -0.51 0.74 

Illustrate facts c prlnc. & theory of lectures/books 0.42 -0.29 0.71 

Apply one's Insights, discoveries & conclusions -0.43 0.27 0.70 

Use mental skills Inherent to professionals 0.55 -0.13 0.68 

Experience kinship with the scientist -2.43 -1.76 0.67 

Design new exp. 1n their own fields -0.58 0.07 0.65 

Appreciate the usual & unusual -0.58 -1.16 0.58 

Work Independently of others -1.57 -1.04 0.53 

Experience spirit & essence of sclent. Inquiry 0.57 0.07 0.50 

Experience joys & sorrows of experimenting -0.71 -1.16 0.45 

Act Independently & take Initiative -0.15 0.27 0.42 

Appreciate relationship between nature 4 science 0.15 -0.24 0.39 

Use motor skills inherent to professionals -0.58 -0.93 0.35 

Do experiments 1.13 1.47 0.34 

Form attitudes related to value & uses of exp. science 0.85 1.18 0.33 

Intuitively understand scientific phenomena -1.14 -1.47 0.33 

Be Interested 1n the subject area 0.15 0.47 0.32 

Use the lab as an Instrument for discovery 0.57 0.87 0.30 

Experience challenge of exp. method -0.15 -0.44 0.29 

Have a critical attitude to exp. results 1.86 2.09 0.23 

Solve problems 1n a critical, academic way 1.48 1.67 0.19 

Plan ahead 0.57 0.76 0.19 

Take active part in the process of science -0.37 -0.55 0.18 

Interpret data 1n literature 0.42 0.56 0.14 

Make decisions 1n proper course of action of prob-solv1ng 1.28 1.18 0.10 

Survey literature relevant to some problem 0.57 0.47 0.10 

Experience past and present scientists 1 joy -0.67 -0.75 0.08 

Deeply understand the discipline studied 1.13 1.07 0.06 



n.b.: AZ 1s an absolute score 



Appendix 15 



Order of magnitude of differences in normalized end-term scores 
between OuN and DUE 



The first column gives the Item muter for the specific end-tenn from the Inventory (appendix 7), 
The roman numerals 1n coluwn two correspond with the general end- terms* 



Endterm 

69 II Do experiments 

37 I Discover limitations of a theory/model 

70 I Work Independently of others 

99 I Use mental skills Inherent to professionals 
49 I Formulate a problem that can be researched 

13 II Illustrate facts, prlnc. & theory of lectures/books 
11 I Work 1n groups to solve sclent, problems 

90 I Survey literature relevant to some problem 
7 I Use motor skills Inherent to professionals 

2 II Use the lab as an Instrument for discovery 
93 II Experience kinship with the scientist 

73 II Experience joys & sorrows of experimenting 

78 II Experience spirit & essence of sclent. Inquiry 
40 I Concretize theoretical notions 

60 II Experience past and present scientists 1 joy 

74 I Interpret data 1n literature 

39 I Act Independently & take initiative 

57 I Apply one's Insights, discoveries & conclusions 

59 II Build framework for facts, prlnc & theory from lect/bocks 

58 I Be self-confident and Independent 

3 II Solve problems 1n a critical, academic way 

66 I Approach a problem with an open mind 
76 II Design new exp. in their own fields 
48 I Plan ahead 

30 II Determine limits under which a theory applies 

6 II Deeply understand the discipline studied 

14 I Hake decisions in proper course of action of prob-solv1ng 
82 I Tackle a problem without help of others 

79 I Have a critical attitude tc exp. results 

36 II Intuitively understand scientific phenomena 

67 I Take active part in the process of science 

81 II Appreciate relationship between nature & science 
51 I Appreciate the usual & unusual 

15 I Form attitudes related to value & uses of exp. science 
96 II Work in research & development labs 

8 II Approach observed phenomena from a sclent, point of view 
27 II Be interested in the subject area 
50 II Experience challenge of exp. method 
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DUE AZ 
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-0.15 
2.16 
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-0.01 
0.15 
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0.72 
1.59 
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-0.88 
-0.15 
-0.01 
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-1.88 
1.88 
0.15 
-0.15 
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0.42 
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0.42 
-0.86 
1.48 
1.43 
-0.58 
0.57 
-0.86 
1.13 
1.28 
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1.86 
-1.14 
-0.37 
0.15 
-0.58 
0.85 
-2.00 
1.86 
0.15 
-0.15 
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0.57 
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0.30 
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•b.: AZ is an absolute score 
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Appendix 16 Ordar of magnitude of differences in normalized end-term scores 
between OuN and UE 



The first column gives the item nuntoer for the specific end- term from the Inventory (appendix 7), 
The reman numerals in column two correspond with the general end- terms. 

End-term OuN UE AZ 

69 II Do experiments -0.72 1.47 2.19- 
96 II Work in research & development labs -1.88 -0.35 1.53 

58 I Be self-confident and independent -0.15 -1.67 1.52 

59 II Build framework for facts, princ & theory from lect/books -0.29 -1.78 1.49 

2 II Use the lab as an instrument for discovery -0.44 0.87 1.31 
90 I Survey literature relevant to some problem 1.59 0.47 1.12 

8 II Approach observed phenomena from a sclent, point of view 1.88 0.98 0.90 

70 I Work independently of others -0.29 -1.04 0.75 
51 I Appreciate the usual & unusual -0.44 -1.16 0.72 

60 II Experience past and present scientists 1 joy -1.45 -0.75 0.70 
7 I Use motor skills inherent to professionals -1.59 -0.93 0.66 

48 I Plan ahead 0.15 0.76 0.61 
74 I Interpret data in literature 1.15 0.56 0.59 

36 II Intuitively understand scientific phenomena -0.88 -1.47 0.59 
82 I Tackle a problem without help of others -0.88 -1.47 0.59 
99 I Use mental skills inherent to professionals -0.72 -0.13 0.59 
30 II Determine limits under which a theory applies -0.44 0.07 0.51 
79 I Have a critical attitude to exp. results 1.59 2.09 0.50 

3 II Solve problems in a critical, academic way 2.16 1.67 0.49 
15 I Form attitudes related to value & uses of exp. science 0.72 1.18 0.46 

13 II Illustrate facts, princ. & theory of lectures/books -0.72 -0.29 0.43 
73 II Experience joys & sorrows of experimenting -1.59 -1.16 0.43 

14 I Make decisions in proper course of action of prob-solving 1.59 1.18 0.41 
67 I Take active part in the process of science -0.15 -0.55 0.40 
78 II Experience spirit & essence of sclent. Inquiry -0.29 0.07 0.36 

6 II Deeply understand the discipline studied 0.72 1.07 0.35 

37 I Discover limitations of a theory/model 0.72 1.07 0.35 

49 I Formulate a problem that can be researched 1.01 0.67 0.34 
27 II Be interested in the subject area 0.15 0.47 0.32 

39 I Act independently & take initiative 0.58 0.27 0.31 
93 II Experience kinship with the scientist -1.45 -1.76 0.31 
66 I Approach a problem with an open mind 0.86 0.56 0.30 

50 II Experience challenge of exp. method -0.15 -0.44 0.29 
81 II Appreciate relationship between nature & science -0.01 -0.24 0.23 
11 I Work in groups to solve sclent, problems -0.01 -0.13 0.12 
76 II Design new exp. in their own fields -0.01 0.07 0.08 

40 I Concretize theoretical notions -0.58 -0.51 0.07 
57 I Apply one's insights, discoveries & conclusions 0.28 0.27 0.01 

n.b.: AZ 1s an absolute score 
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